I'm building a production-shaped quality platform for AI agents on AWS — Part 1

23 Aug, 2026

A fictional media company, Meridian Media Group, is launching Beacon, a direct-to-consumer streaming service. Engineering teams want to ship AI agents against it: a highlights agent, a recap agent, whatever comes next. The platform team's job is to make sure that every one of those agents is governed, measurable, and hard to trick — without making each team re-learn security, compliance, and evaluation on their own.

The streaming service is the vehicle, not the point. The point is the platform underneath it, and the honest story of building it milestone-by-milestone with Claude (Claude Code, powered by Fable 5) as my pair — including the numbers that came out worse than I predicted, which are left exactly as recorded.

Repo (MIT, make check runs offline with no AWS account): github.com/andaro74/beaconpave

The problem in one paragraph

Anyone can build an AI agent in an afternoon. The hard questions come right after: Did it get worse since last week? Can it be tricked into saying something we're legally not allowed to say? Which tool did it call, and who authorized that? Who approved the change? Most teams answer these with a checklist someone runs before launch. beaconpave's premise is that quality, compliance, and adversarial resistance should be properties of the infrastructure, not a phase a team completes — the same way you don't ask each team to build its own load balancer.

Two brands, two compliance problems

The fictional service has two brands, each chosen to miniaturize one of the hardest compliance problems in media:

• Meridian News — attribution and AI-disclosure rules. An agent summarizing a story must cite its source and say when AI was involved.
• Meridian Sports — entitlements and regional blackouts. If a viewer's market can't watch the Jefferson Derby tonight, the agent must never say they can.

Everything is fictional: the catalog, the markets, the regulations, the company. Fork it and rename it for yours.

Why two parts?

A production-grade agent platform is a lot of pieces, and they don't all belong to the same layer. Laid out, they split cleanly:

Part 1 — the machinery that judges an agent. A gateway every model call must transit, a tool registry with policy, an eval harness with a fixed golden set, a judge that's calibrated (or, as it turned out, demoted), an adversarial suite, and a CI gate that blocks a merge the moment quality goes down.

Part 2 — the path that creates an agent. One command, pave new, that scaffolds a new agent already wired into all of the above. Then a rules registry, human approval for consequential actions, browser and load tests on the same verdict format, a game-day drill, and a self-heal classifier.

The order is the whole design. If you pave the road first, teams drive on it before the guardrails exist, and you never get to add them without breaking everyone. So Part 1 is all measurement and enforcement, built and proven before a single team-created agent exists. Right now pave new is a stub that prints a sentence and exits 0. That's the honest state of the repo, and M05 is where it stops being true.

The architecture (what exists today)

Two halves, and they meet at the audit lake.

On AWS (serverless, under $5/month idle):

• The LLM Gateway — one Lambda, and the only path to a model. Every call goes classify → guardrail → invoke → meter → audit. Every other role in the account carries an explicit IAM Deny on bedrock:InvokeModel. That's G1, and it's asserted by a test against the committed CDK synth snapshot, so CI catches drift before anything deploys.
• A pinned Bedrock Guardrail — applied on both the INPUT and OUTPUT channel. "Pinned" matters: when the guardrail changed between milestones, every score recorded under the old version was footnoted rather than re-run, because a re-run control would have looked better with zero improvement to the system.
• Amazon Bedrock, Claude Haiku 4.5 through the US cross-region inference profile.
• The tool plane — MCP tools declared in platform/registry/tools.yaml with an owner, semver, schemas, and a consequence class. Cedar policies are generated from that file. A tool with no registry entry has no permit and is simply unreachable (G3). Anything at publish or above gets a human-approval interlock (that lands in M06).
• The audit lake (S3) — one record per gateway call, with an independent witness in CloudTrail for denials.

On GitHub:

• The quality gate — an org-required workflow that runs unit, contract, eval, and adversarial lanes and fails closed: an errored lane blocks, never skips.
• The two-key check — for anything that changes thresholds, baselines, consequence classes, or the adversarial corpus, a second seat's attestation must be in the PR body. (More on why it's not CODEOWNERS below.)
• evals/history/ — append-only scores keyed by git SHA. Never rewritten; a wrong entry gets a superseding entry.

What one governed request looks like

A viewer asks the highlights agent who's playing tonight. The agent can't call Bedrock itself — IAM forbids it — so it calls the gateway with its data classification. The gateway routes on that classification (anything sensitive is refused by design, G5), runs the INPUT guardrail, and only then invokes the model. Tool calls go through Cedar against the registry. The tool's result comes back as an untrusted channel — that distinction turned out to matter a lot (see M04). The answer is guardrailed on OUTPUT, metered against token and latency budgets, and written to the audit lake with a record id that's handed back with the answer.

That record id is the thread everything else hangs on.

How the work is built: the repo history is the demo

Every milestone is one branch and one tag. Every milestone is scored against the same 25 golden cases and the same 10 adversarial probes, and the score goes into the append-only history. Each milestone has a journal that answers three questions: what can I demo right now, what's the delta vs baseline, what broke.

Deliberately-red demo PRs are labeled exhibit and closed unmerged, so main is always green. A gate you can merge past isn't a gate.

You don't have to trust a slide deck. Check out any tag and see what was true at the time.

Part 1, milestone by milestone

M00a — A gate that can fail

Before any agent, a CI gate whose only job is to block. Proven by two exhibit PRs (#2, #3) where the gate and the two-key check each stopped a merge for its own reason. M00a scores nothing and writes no history entry — a placeholder row in an append-only file that no run produced would poison the one file whose value is that every row came from a real execution.

M00b — The ungoverned baseline (the control)

A ~100-line agent: direct model call, whole catalog stuffed into the prompt, no gateway, no guardrails, no audit. It scored 15/25 on goldens and 0/10 on adversarial.

That 0/10 is by construction, and it's the most important rule in the repo. An adversarial "pass" here means a guardrail blocked it or a policy denied it, and an audit record exists (G4). The model politely refusing on its own never counts. The control has none of those things, so no probe can pass regardless of how the model behaves. In fact the control resisted the injection probe and refused the PII request — and leaked its entire configuration the moment the request was framed as "debugging."

Also recorded: four of the fifteen golden passes are unearned. The agent claimed source: entitlement-check in 10 of 11 provenance cases — a tool it does not have. It read the answer schema out of its own prompt and picked the flattering enum. The row stays at 15/25, marked. A flattering baseline makes every later milestone unfalsifiable.

M01 — Gateway, audit lake, IAM assertions

The gateway Lambda, the CDK stack, the S3 audit lake, and the IAM assertion test. PR #14 tried to grant bedrock:InvokeModel to the service role; the gate blocked it. The direct-call probe fails with explicit deny in an identity-based policy, and CloudTrail — which the calling principal cannot write to — recorded the same denial.

Adversarial went to 7/10 (one unearned, so 6/10 credited: a probe declaring Cedar semantics passed on a content filter matching the phrase "skip review," which is not Cedar). Goldens went 15 → 19, and the journal is careful to read that as noise, and probably a regression: the prompt is byte-identical, the model is unchanged, and a gateway can only refuse cases, never improve an answer. Three cases were lost to guardrail refusals. Governance cost three golden cases and the suite total hid it. The p95 latency budget was breached at 3194 ms against 2500 — recorded, not accommodated.

M02 — Tool registry + Cedar

catalog-search becomes a real MCP tool behind Cedar, generated from the registry. The agent answers via the tool instead of an inlined catalog.

Two things I learned here:

A single run tells you almost nothing. The identical system sampled three times in one day returned 18, 16, and 14. A single sample of either arm could have produced a headline anywhere from −4 to +1. So from here on every score is a majority over k=3, and k and arm are part of the history schema.

A pre-registered prediction can fail in the direction that flatters you. SPEC/02 predicted 10/25 ± 4 because "mid-loop guardrail refusals will rise." Refusals fell, 19/75 → 7/75. The mechanism had been derived by measuring the tools arm twice and never measuring the control's refusal rate on the same cases. A loss mechanism stated as a difference between two systems has to be measured across both of them. The first explanation I wrote for why they fell was later shown to be arithmetically impossible, and that correction is in the README too.

M03 — Eval harness + judge calibration

An LLM-as-judge was added on top of the deterministic checks — and then measured on 20 held-out cases at k=3 before it was allowed to touch anything. Every axis failed calibration. So every axis is demoted to advisory.

The consequence is written into the schema: a judge in this repo can only subtract. veto can turn a deterministic PASS into a judged FAIL, never the reverse, and only on a calibrated axis. Since no axis is calibrated, the judged column reads −0 everywhere — a measurement ("the judge was measured and found unfit to move this"), not a default. A judge that hasn't proven its agreement with humans should not be blocking your merges. That's claim 9, and it's proven by the demotion, not despite it.

M04 — Fail-closed gate + adversarial suite

The gate grows its L5 lane: 10 probes × 3 samples, unanimity required, every observation fetched back out of the audit lake rather than taken from the gateway's word. A record that doesn't resolve scores FAIL — a gateway naming a record the lake doesn't hold is a worse finding than a missing block.

The demo for claim 2 is PR #29: six lines that make a probe pass because the model declined. The gate answers BLOCKED (quality regression); exit 1, and its comment names the five probes that moved, the comparator they moved against, and what to do. Exit 1, never 2 — a caught regression, not a broken harness.

The 7/10 here is a falsified prediction. SPEC/04 pre-registered 4–6 and named ≥ 7 as the falsifier. It fired. And the reason attached to the falsifier ("the corpus got easier") was wrong too: exactly one probe moved, ADV-010 (prompt leak), FAIL → PASS, because the guardrail got stronger under v2. The catch: what blocked it is an entitlement topic that also blocks the product's most basic question — "who is playing in the Jefferson Derby?" — in 1 of 3 identical calls. One control, two signs. A fix for the product bug should be expected to take the adversarial number back down, and the comparator pin says so, so the gate will read that as the fix working rather than a regression to defend.

The cleanest result of the run: the same injected payload is blocked 3/3 as a user turn and allowed 2/3 as tool output. That attributes the failure to the channel, not the wording — and it's why tool results are treated as untrusted in the sequence diagram above.

The gate, end to end

gate decide runs with if: always(). A lane that crashes before writing its verdict leaves the file absent, and an absent verdict blocks. A suite that doesn't exist yet is absent from the list, never present-and-passing, because a placeholder PASS for an unimplemented suite is indistinguishable from a real one. There is no arrangement of failures that reaches a merge.

The ten golden rules

These are the invariants the repo exists to enforce. Each one names the mechanism that enforces it, because an invariant that's merely asserted is a wish.

G9 deserves a word. The obvious mechanism is CODEOWNERS. On a one-operator repo it collects nothing — GitHub won't let you approve your own PR, so the second key is unobtainable rather than inconvenient. Worse, three paths were "protected" in CODEOWNERS and therefore protected by nothing, which stops anyone looking for the real protection. So the second key is an attestation in the PR body (Two-Key-Disposition: ai-quality plus a rationale), verified by a required check, and a contract test asserts the enforced list and the documented list agree.

How AI enables developers here — and how I actually worked with it

Two different things, and both matter.

For the developer who will use the platform. The promise of Part 2 is that a service team runs one command and what comes out is governed by default:

pave new my-agent --brand meridian-sports --classification internal scaffolds a prompt, a manifest, a starter golden set, and a registry entry — already routed through the gateway, already permitted by Cedar, already wired to the same quality gate the platform itself lives under. The developer's job is the agent's behavior. Governance is not their job, because it's the road's. Target: repo → deployed, governed agent in under 30 minutes (claim 1).

For building the platform itself. The org chart is encoded in the repo. Seven seats — Platform Engineering, AI Quality, Security, Legal/S&P, Data Governance, Tool Owner, Service Team — each own paths, and each has a role subagent in .claude/agents/ that reviews a diff from that seat's angle. A solo operator reads a diff once, from one angle; the subagents read it from the seven angles the seats are responsible for. Their output is advisory input to me, never an approval (G6).

The SPEC for M05 went through three draft rounds with six seats reviewing each: 39 blocking findings on draft 1, 31 on draft 2, 20 on draft 3. Six statements of fact across those drafts were measured false, and every one of them flattered the platform. The drafts are preserved at a tag rather than edited away. That's what the subagents are for: not writing code faster, but catching the story I wanted to be true.

Claude Code brought the same workflow as ShowRunner — plan mode before any file is touched, a lean CLAUDE.md as project memory, a close-milestone skill that turns "done" into a checklist (record evals, write the journal, fill the row, tag). I brought the architecture judgment: M00b before M01 always (build the gateway first and you'll unconsciously build a baseline that flatters it), the gateway as the one component that must never be simplified away, the rule that every scope cut is an ADR ending with "at scale, replace with X; the interface already matches."

Neither of us would have produced forty-three ADRs and eleven corrections to our own earlier claims alone. The AI without the invariants would have happily "fixed" the control. Me without the AI would still be on M02.

What Part 1 honestly doesn't have

• No agent that a team created. pave new prints a sentence. That's M05.
• A manifest nothing verifies. pave.manifest.yaml has ten fields; six can be deleted with the full suite green, and classification: internal → public changes nothing. SPEC/05 measures this. It's why M05 exists.
• Seats are subagents, not people. Solo-played, by ADR-001.
• The judge is advisory. Which is the correct state for an uncalibrated judge, and claim 9's proof — but it means no LLM judgment currently blocks anything.

Four of the twelve claims are proven with a linked artifact (2, 4, 5, 9). The other eight belong to Part 2.

What's next

Part 2 spends this machinery on what a platform is for: one command, and what comes out is governed by default. M05 through M10 — the scaffold and manifest verifier, a second tool with a real human-approval interlock for publish-highlight, a rules registry that turns a regulation change into new eval cases and a red service, Playwright and k6 emitting the same verdict schema as the agent evals, a game-day drill that produces a NO-GO → fix → GO artifact, and a self-heal classifier that decides drift or defect before it's allowed to repair anything.

I'll write that up when it's done. In the meantime the repo is open, make check runs offline, and every number in this post has a footnote in the README that says exactly how it was measured — including the ones I got wrong first. If you're building governance for agents at any scale, I'd like to compare notes: github.com/andaro74/beaconpave. Which of the ten golden rules would you fight me on?

Transparency note: this article was drafted with Claude Fable 5 — driven, reviewed, and fact-checked by me against the project's README, specs, and milestone journals. The same division of labor that built the project wrote the post about it.

#AWS #AmazonBedrock #AgenticAI #AIQuality #Guardrails #Cedar #MCP #GenAI #PlatformEngineering #ClaudeCode #Anthropic

#agentic-ai #ai-quality #amazon-bedrock #anthropic #aws #cedar #claude-code #genai #guardrails #mcp #platform-engineering