I Built a Miniature Production Demo of an Agentic Regulatory-Change Assistant

23 Aug, 2026

Lessons from RegDelta — for compliance and operations leaders learning what "AI with citations" actually means in code, and for engineering teams deciding how to prove an agent is better than the chatbot it replaces.

Most RAG demos prove that a model can find a relevant paragraph. Almost none prove that it got the date right — and in regulatory work, the date is the whole answer. A delayed effective date looks exactly like a delayed compliance deadline to a similarity search. It is not the same thing, and the difference is measured in months of exposure.

So I built the assistant that knows the difference — a deliberately small one, with production shape in every layer — and called it RegDelta: an agentic FDA regulatory-change assistant on AWS that answers "what changed, does it apply to us, and what's the real deadline?" with a Federal Register citation on every claim, and that refuses to answer when it isn't sure.

The graph decides the dates. The gate decides what "correct" means.

Twenty days, August 3 to August 22. Eight milestones, each one a tagged commit scored against the same fixed golden question set. 223 commits. 15 architecture decision records, written the day each trade was made. Roughly 900 unit tests. A corpus that grew from 4 Federal Register documents to 52 on its own, by a scheduled poller. And one honest number, stated the same way in every milestone: naive RAG scores 4/20; the agent scores 18/20 — on both retrieval tiers, identically.

This production demo was designed and architected by me, Hector Flores, Principal Software Architect, and implemented by Claude Code using Claude Opus 5 and Fable 5. I decided what correct means; Claude wrote the code; the gates held both of us to it.

Everything below is public and verifiable. The full source, the per-milestone evidence packs, the eval history keyed by git SHA, and the ADRs are on GitHub: https://github.com/andaro74/regdelta — the live demo is at https://d2rdgeiujg622n.cloudfront.net — and RegDelta joins the case-study series on my site, alongside AgentPave, the Agentic PII Erasure Platform, and ShowRunner: https://floresinnovations.com/projects

The use case: the question that fools everyone

The demo's signature question is one sentence: "The effective date of the new 'healthy' claim rule was delayed. Did the compliance deadline change?"

In December 2024 FDA finalized the updated "healthy" implied nutrient content claim, 21 CFR 101.65(d), with an effective date of February 25, 2025 and a compliance date of February 25, 2028. In February 2025 a second document delayed the effective date to April 28, 2025. A naive reader — human or RAG — slides the compliance deadline forward with it. It did not move. The delaying document says, in terms, that the compliance date remains unchanged, and RegDelta's answer quotes that sentence back, cites both documents, and reports a confidence of 0.97.

The product surface is deliberately small: a verdict table (product, trigger, required change, real deadline, confidence, citations), a free-text question box, a handful of canned scenarios for a fictional company called Nordvale Foods, and four instrument tiles that report which tier answered, whether it was cached, and how long retrieval took. The assistant was never the point. The point is that every number on that screen is either read from a recorded artifact or linked to the sentence on federalregister.gov that proves it.

That framing matters for anyone evaluating an AI compliance tool right now. The question is rarely "can it summarize a rule?" It can. The question is: when the rule is amended on a Tuesday, does the system know which of its three dates moved — and can it show you the sentence?

The architecture

Every component has a single job. A scheduled poller checks the Federal Register daily and writes to an S3 corpus bucket that is the sole source of truth — raw, parsed, chunked, with embeddings computed once at ingest and never again. Two retrieval tiers hydrate from it: S3 Vectors, always on and roughly $2 a month idle, and OpenSearch Serverless, an ephemeral hot tier that exists only between make up and make down. A LangGraph agent on Bedrock reads from whichever tier is active, plus a DynamoDB amendment graph, and serves a verdict through FastAPI on Lambda behind a static UI on CloudFront.

Two design decisions carry most of the quality, and both are enforced rather than asserted.

Timeline questions never touch vector search. Effective dates, compliance dates, stays and supersessions are answered from typed edges in the amendment graph — SUPERSEDES and CONFIRMS rows scoped to the specific field they affect, extracted at ingest. The "healthy" claim answer above is read from two of those rows. One document moved the effective date and confirmed the compliance date, and those are two separate facts stored as two separate rows. Nobody asks a language model what a paragraph feels like.

Below 0.7 confidence, it refuses to answer. "Are we affected?" with no company profile attached renders NEEDS HUMAN REVIEW and mints a resume token instead of guessing. In compliance, confidently wrong is worse than slow.

And one phrasing rule the repo enforces on itself, in CLAUDE.md and in the diagram's own comments: the second tier is the same retrieval algorithm on different infrastructure. Not hybrid — hybrid measured worse than vector-only, 7/9 against 9/9, and is off by default (ADR-0009). Not faster — that justification was measured at 889 ms median against 354 ms and retired (ADR-0012). Its remaining case is concurrency, and a later milestone measured that too: at 50 calls per second the hot tier's p95 was 186 ms against 281 ms, so it stays, bounded to the band where it was measured (ADR-0014).

That last pattern is the one I'd put on a slide for any executive team: a justification that was never measured is a belief. The repo is only allowed to keep beliefs that have a reversal condition written beside them.

The score, stated the honest way

The progression is the argument: a naive-RAG baseline that scores 30% is the control; ingestion builds the amendment graph; two-tier retrieval gets recall to 1.0 on both tiers; the agent graph fixes the four trap questions; and then API, deployment, load testing and governance make it production-shaped without moving the score. M04, M05, M06 and M07 all report 18/20. That is the correct result. A governance milestone reporting a score improvement would be reporting either a coincidence or a gate that had started grading itself.

The two misses, q12 and q15, are documented defects with triage write-ups — recorded, not rounded away. The control is never "improved" (ADR-0002); it is re-run at every milestone's own commit, over that milestone's corpus, carrying a corpus fingerprint so the comparison is real. When the corpus grew from 34 to 49 documents during a closing session, that fingerprint is why anyone could tell.

The demo in three acts

Act 1 — the money shot. Click the "healthy" claim scenario with the cache bypassed. About twelve seconds later — "it's reading the actual Federal Register documents" — the verdict table fills in with the un-moved 2028 deadline and both citations. Untick bypass, ask again to populate the cache, ask a third time: back in well under a second, labelled hit, with the tier and latency tiles greyed out and marked as provenance of the stored answer, because nothing was retrieved for this request. That is the cost story, told honestly.

Act 2 — when it refuses. Click "Are we affected?" The question carries no product and no claim, so there is no honest answer for this asker. The page renders NEEDS HUMAN REVIEW with the reason, and notes that a resume capability was minted — without printing the token.

Act 3 — one change, three doors. This is the governance demo, and it's the half I think matters more. The org chart is encoded in the repository: CODEOWNERS maps files to role seats — PM owns the specs, the compliance SME owns golden ground truth, Security owns tool policy and infrastructure — and role subagents in Claude Code run first-pass review from each seat. The premise on screen: a golden-set question is failing, and a pull request wants to change what "correct" means.

Door 1: an engineer does the obvious thing and adds an accept token to the golden set so the build goes green. A required check called ruling-cited blocks the merge and says why in its own words — engineering may not decide what correct means. No bypass is offered, because the admin bypass was removed and the repository owner is subject to the rule like anyone else. (The token in question was also a real false pass: the model's answer read "not a fair reading," and "fair reading" is a substring of it.)

Door 2: the right path. Run the sme-eval-triage agent, land the SME ruling as its own pull request, then land the change citing the ruling. The check goes green and names the document. Two pull requests, because the decision and the code are two different acts.

Door 3: an IAM policy "helpfully" widened to resources: ["*"] while fixing something else. The security-reviewer agent returns a HIGH finding with a file and line; the unit check is red as well; the diff dies in review. The engineer owns every line the AI wrote, and the system is built assuming some of those lines will be wrong.

All three doors were run against real pull requests — #20, #17 → #18, and #21 — and the evidence sits in the repo.

The lessons — and this is the part worth your time

The recurring defect in this project was never wrong code. It was an instrument that measured something adjacent to the claim and read exactly like one that measured the claim. The demo page's /health endpoint reported what the system was configured to, not what had answered — so a cold Lambda served the wrong tier for its first sixty seconds and the one endpoint whose job was to notice was structurally unable to. A cache hit returned the stored body, so the tier badge described a request from up to an hour earlier, possibly on the other tier; a hot-tier scorecard once read 5/5 having reached the hot tier zero times. A client stopwatch under the words "retrieval latency" was measuring generation. Eleven instances in one milestone, a twelfth found by writing the rule down, and the rule is now an ADR: an instrument reads the field that describes its own claim (ADR-0013).

The second lesson is about gates that protect honesty and then quietly break it. A guard added to stop a scorecard reading cached answers rejected any response without a cache state — and the offline shim used by the baseline emitted none. For three days nothing noticed, because nothing re-ran the baseline. A guard written to protect card honesty had silently disabled the one card every other claim is a delta against. The fix was small; the lesson is that the control needs its own alarm.

The third is the one that surprised me. The eval gate had been defined as passed == total — and that bar had never been satisfied by any recorded run in the project's history. A gate nobody can pass is not a gate; it is a reason to add bypasses. The PM seat ruled that the bar was wrong, the gate now fails on a regression against recorded history, and the milestone that had been stuck for two months on one intermittently flaky question closed without admitting, exempting or rescoring it. Then the new gate immediately caught a real defect in its own milestone's work: a documentation-only pull request was blocked twice on q05, diagnosed as flake, then non-determinism, then Bedrock load, then completion length — all four wrong. The metrics showed the verdict call had succeeded and a JSON parser had failed on a complete reply, collapsing the answer to empty at confidence 0.00. The gate was right; the product was not.

For compliance and operations leaders, translate it this way: when a vendor shows you a dashboard, ask what field each tile is reading. If the answer is "the configuration" rather than "the request," you are looking at /health.

On cost: each uncached question is about 5,900 Opus tokens and twelve seconds, against a daily account cap that cannot be raised. Every pull request runs the full golden set at roughly $0.20, which is exactly the price that makes "re-run until green" a visible behaviour rather than a free one — the repo names that as a thing to watch for. The always-on tier idles at about $2 a month; the hot tier bills about $0.24 an hour only while it exists, and a janitor tears it down. Nothing bills while idle.

How I actually worked with AI

The division of labor was explicit, and it is encoded in the governance model rather than described in a README paragraph.

I designed and architected the project as Principal Software Architect, and held every seat. The architecture, the two design rules, the milestone sequence, the baseline-first discipline, and every seat ruling — which accept tokens are admitted, which false fails are acknowledged, when a justification is retired — were human judgment calls, each written down with the evidence a reader could use to prove it wrong. There is one human in this repository, and ADR-0005 says what that means honestly: the signature is theater, the seat is not. What makes a ruling sound is that it can be falsified against a file in git.

Claude Code implemented it against that spec — on Claude Opus 5 and Fable 5 — working from a standing CLAUDE.md contract, one milestone per session, each SPEC's "Done when" as the exit criterion, a /close-milestone skill that assembles the evidence pack and records the eval, and role subagents (sme-eval-triage, security-reviewer, eng-code-reviewer, pm-spec-reviewer) that the rules require before a pull request opens. The one standing instruction that matters most: never edit the golden questions to make a failure pass. Run triage and stop for a human.

The commit history is the supporting evidence: 223 commits, docs-first, each milestone closing only when the golden set passed on the deployed API and the deviations were written up the same day.

And the most important moment was when the gate blocked me. Door 1 wasn't a staged strawman. The admin bypass was removed on August 22, the required checks bound the owner from that moment, and the first thing they did was refuse a documentation-only change on a defect I would have otherwise shipped. Propose-and-dispose with humans at the boundaries is not a compliance formality. It is where the defects get found.

The takeaway

If you lead a compliance or operations function: the citation, not the summary, is where your AI investment compounds — and an assistant that can say "needs human review" is worth more than one that always has an answer.

If you run an engineering team: clone it. Read milestones/M*/README.md for what each one could demo and what broke. Read the footnotes under the progression table before quoting a row; they are load-bearing. Every trade-off has an ADR with a reversal condition beside it.

The graph decides the dates. The gate decides what "correct" means.

RegDelta was designed and architected by Hector Flores, Principal Software Architect, and implemented by Claude Code on Claude Opus 5 and Fable 5 — a collaboration of architect and AI, with the division of labor encoded in the governance model above.

🔗 GitHub repository (clone it, read the ADRs, run make evals): https://github.com/andaro74/regdelta 🔗 Live demo: https://d2rdgeiujg622n.cloudfront.net 🔗 Full case-study series on my website: https://floresinnovations.com/projects

Building agentic systems for regulated work? Let's compare notes.

#AgenticAI #RegTech #AWS #ClaudeCode #RAG #LangGraph #Bedrock #QualityEngineering #Governance #FDA

#agentic-ai #aws #claude-code #regtech #quality-engineering

#agentic-ai #aws #claude-code #quality-engineering #regtech