I Built a Miniature Production Demo of an Agentic AI Platform in Seven Days
Lessons from AgentPave — for media executives learning what "AI governance" actually looks like in code, and for platform teams deciding what to build first.
Most agentic AI demos prove that an agent can work. Almost none prove that an organization of agents can work — that a second, third, and tenth team could ship governed, evaluated, observable agents without rebuilding the same machinery every time.
That is a platform problem. So I built the platform — a deliberately miniature one, with production shape in every component — and called it AgentPave: an agentic AI developer platform on AWS where one command scaffolds a governed agent that arrives with evals, guardrails, tracing, and a failing-closed CI quality gate already attached.
The paved road provides. The quality gate decides.
Seven days. Seven milestones, one per day. 120 commits, each one evidence of what was built when. 38 architecture decision records, written the day each trade was made. 736 hermetic tests. Six CloudFormation stacks. And $20.45 of total Bedrock spend — measured by the platform's own metering, not reconstructed from an invoice.
Everything below is public and verifiable. The full source, commit history, and ADRs are on GitHub: https://github.com/andaro74/agentpave — and AgentPave joins the case-study series on my site, alongside the Agentic PII Erasure Platform and ShowRunner: https://floresinnovations.com/projects
The use case: a Streaming Catalog Concierge
If you run a media business, the sample service riding this platform will look familiar: a single agent that answers questions about TV shows and schedules — what network airs a show and when the next episode lands, two-sentence summaries with genres, which titles are currently running, and a structured enrichment mode that returns clean JSON metadata for a catalog record.
It's grounded in the free TVMaze API, exposed as an MCP tool. The concierge is deliberately small — four capabilities, one tool — because the concierge was never the point. The point is everything wrapped around it: the scaffolding, the gates, the guardrails, the meter, the traces. The app is the passenger; the platform is the vehicle.
That framing matters for media executives evaluating AI initiatives right now. The question is rarely "can we build an agent that recommends content or answers catalog questions?" You can. The question is: when the tenth team wants one, does each build take a quarter — or a command? And when someone changes a prompt on Friday afternoon, what stops a quality regression from reaching your audience?
The architecture
The design gives every component a single job. A pave CLI scaffolds a new service into its own stack with the platform SDK pre-wired. Every model call goes through a central LLM gateway — Bedrock behind it, guardrails applied centrally, data-classification routing, and per-service token metering into DynamoDB. Tools live in a registry with owners, versions, JSON schemas, and a consequence class, with Cedar policy binding each agent identity to exactly the tools it may call. An eval service runs a golden dataset with deterministic asserts plus an LLM-as-judge, and everything is traced end to end with OpenTelemetry into a dashboard built as code.
Five invariants hold it together, and each is enforced, not asserted: every model call goes through the gateway (no service holds Bedrock permissions of its own — checked by IAM assertions at synth time); quality gates fail closed (a gate that errors blocks, never skips); nothing bills while idle; the local check suite is fully hermetic; and adversarial probes pass only when the platform blocked the attack — never because "the model resisted."
That last invariant is the one I'd put on a slide for any executive team: safety that depends on a model's good manners is not a control. Safety enforced by infrastructure is.
The demo in three acts
Act 1 — the paved road: pave new catalog-agent and minutes later a deployed agent answers a grounded question, blocks a prompt-injection attempt at the guardrail layer, meters the request at $0.000629, and emits traced spans. All five claims are deployed facts, not mocks.
Act 2 — the gate bites: a pull request changes the agent's prompt to be drastically more concise. The eval gate blocks the merge at 29/31, −6.5%, and posts the score diff as a PR comment. A quality regression caught by infrastructure, not by a viewer. The red pull request stays open in the repo's history on purpose — it is the evidence.
Act 3 — self-healing, human-triggered: a tool schema change breaks a contract test; a classifier decides the failure is drift rather than a defect; a human runs Claude Code against that verdict; it proposes the repair as an ai-proposed PR; a second human approves. AI maintaining the platform's own QA — under propose-and-dispose, with people at both ends.
The lessons — and this is the part worth your time
Every defect this project found in seven days was found by one of three things: deploying it, changing something of a shape the system had never seen, or a human reading the output. Not one was found by adding another test of the kind already there.
Eleven times across six milestones, a green gate was sitting on top of a check that was measuring nothing at all. The recurring defect was never broken code. It was checks that could not fail — assertions that read the same source on both sides and were therefore incapable of disagreeing with themselves. A spend panel whose every clause was correct and whose column rendered empty. A shadow-eval header that described the configuration rather than the run, and reported "safe to adopt" after comparing a model to itself. Even the architecture diagram: verified twice against a PNG export, while the one surface it existed for — GitHub's markdown renderer — stripped every label out.
For platform teams, the corollary is uncomfortable and useful: a test written by the same person who wrote the check inherits its blind spot. The full log of all eleven, with the fix and the test that kills each one, is in the repo's VALIDATION.md.
For media executives, translate it this way: when a vendor or an internal team shows you a dashboard full of green, the question to ask is not "is it green?" but "what would make it red?" If nobody can answer crisply, you may be looking at eleven of these.
Two more numbers worth pausing on. The QA machinery cost roughly 6,500× the product it was grading ($20.45 of platform against $0.0031 of served answers), and the LLM-as-judge alone was 69% of all spend. At demo scale that's the correct trade — the thesis is that the gate is the expensive part and worth it. At production scale it's the first thing you'd optimize, and the repo names the handles.
How I actually worked with AI
The division of labor was explicit, and I think it's the honest template for this era of building.
I set the use case and established the architecture. The thesis, the five invariants, the scope cuts, the decision that self-healing would be human-triggered rather than headless in CI (because a credentialed AI agent with pull-request write access and a prompt-injection path through the repo it reads was the wrong trade for one demo act — that's ADR-035) — those were human judgment calls, made before code.
Claude Code built alongside me against that spec — mainly Claude Opus, with Fable 5 on select milestones — working from a standing CLAUDE.md contract, custom skills for ADR authoring and eval-case drafting, plan mode per milestone, and the project's own MCP server registered in the assistant's configuration, so the same governed tool served both the production agent and the developer.
The supporting evidence is the commit history itself: 120 commits, docs-first, one milestone a day, each closing only when a hermetic gate and a human-run deployed gate both passed and the deviations were written up as ADRs the same day.
And the most important moment of the week was the one where the AI did not fix the real finding. During Act 3, the repair Claude proposed made the suite green — but a human reviewer noticed the assertion it satisfied was vacuous, and fixing that was exactly the move the AI's license excluded. A more permissive setup would have closed the test and buried the finding in one commit. Propose-and-dispose, with humans at the boundaries, is not a compliance formality. It's where the learning happens.
The takeaway
If you lead a media organization: the paved road, not the agent, is where your AI investment compounds — and quality gates are the difference between "we shipped an agent" and "we can keep shipping agents."
If you run a platform team: clone it. make check needs no AWS account, everything that touches the cloud sits behind an explicit verb, and every trade-off has an ADR with its name on it.
The paved road provides. The quality gate decides.
🔗 GitHub repository (clone it, run make check, read the ADRs): https://github.com/andaro74/agentpave
🔗 Full case-study series on my website: https://floresinnovations.com/projects
Building agentic systems in your organization? Let's compare notes.
#AgenticAI #PlatformEngineering #AWS #ClaudeCode #QualityEngineering #MediaTech #LLMOps #MCP