The Audit Trail Is the Product

In May 2025, the White House published a major report on childhood health. Reporters at NOTUS went through the bibliography and found seven studies that do not exist.[1] One listed author, an epidemiologist at Columbia, said plainly that the paper wasn't real. The report got quietly updated. The press office called it a formatting issue.

A week later, on June 2, the FDA launched Elsa, its own internal AI assistant. The press release said the agency was already using it to speed up clinical protocol reviews.[2]

Seven weeks after that, FDA staff were telling CNN that Elsa cites studies that do not exist.[3] One reviewer summed it up: "Anything that you don't have time to double check is unreliable. It hallucinates confidently."

Both stories ended the same way: a person caught it. Reporters checked the citations in one case, FDA reviewers in the other, since checking is what the FDA does all day. The safety net held.

But that's also the problem. If you have to recheck every line an AI produces by hand, it isn't saving you time. It's generating homework. A reviewer who has to hunt down each citation Elsa gives her is often better off just reading the source herself.

I build AI for clinical trials, and this is the question I spend most of my time on: not how to make the model write better, but how to make its output cheap to verify.

Why we all keep trying

Clinical trials are drowning in documents. A single substantial amendment to a Phase III protocol costs a median of $535,000 to implement and adds about three unplanned months, per Tufts.[4] Most trials have several. The FDA has seen more than 500 drug submissions with AI components since 2016.[5] This is happening with or without anyone's blessing.

In my corner of the industry, the job is turning a protocol into the study build: the electronic case report forms (eCRFs), edit checks, and visit schedules that clinical data managers spend weeks transcribing by hand. The first time I watched a model pull a clean eligibility table out of a 200 page protocol in seconds, I understood the temptation completely. The models are genuinely great at this work.

As drafters. Never as the record.

What the model is actually doing

A language model predicts the next word. It's trained to sound plausible, not to be right, and nothing in its output tells you which you got. Last year researchers had GPT-4o write literature reviews, then checked every reference. A fifth were invented. Nearly half of the real ones had errors.[6]

But fake citations are the failure everybody already knows. Here's the one that actually keeps me up:

That's a check from our pipeline. The model cited the correct sentence and still retyped the number wrong. In a dose field, that's a tenfold overdose. We catch it because every number has to appear in the evidence behind it. Without that rule, it sails through looking perfect.

Two more things, quickly. These systems are not deterministic, even at temperature zero, and vendors swap models behind the same API name whenever they want.[7] You cannot validate a system that will not repeat itself. And when a model outputs "confidence": 0.95, that is not a measurement. It's one more predicted token.

The bar

Regulators saw all of this coming decades ago. Every record in a trial has to meet ALCOA+, the data integrity principles inspectors actually use.[8] Score raw model output against it:

Zero for nine.

On top of that, 21 CFR Part 11 requires an audit trail on every record: who, what, when, why, with old values preserved.[9] And the FDA's 2025 AI guidance, underneath all its framework language, asks one question: can you prove this output is credible for this exact use, and can someone else check your work?[10]

A raw API call fails all of it. GPT-6 will not change that. It isn't a model problem. It's an architecture problem.

What we built, mostly by getting burned

Everything we built sits in one layer between the model and the record, and it exists to make a single sentence true: every value recommended arrives with the evidence it came from. If the system can't show you the sentence, you never see the value.

To be clear about what that means: I can't promise a language model never hallucinates. Nobody honest can. What I can promise is that a hallucination can't reach the record, because one hundred percent of what gets recommended carries source evidence that code has checked and a human has seen. That's not a statistic we measured. It's a rule the architecture enforces.

The model never writes to the record. It proposes. Code decides. Everything the model produces is treated as a claim until deterministic, versioned code verifies it.

I got religious about this the embarrassing way. Early on I found that our output validator was a stub. It returned valid: true for everything. Tests were green. Life was good. That bug taught me more than any guidance document ever has: the scary state isn't a failing check, it's a check that isn't checking. I suspect half the "AI powered" tools in this industry are that stub with a nicer landing page.

Every model call is also recorded in a replay ledger. Same document, same code: identical output, byte for byte, years later. If a prompt or the code changes, old results rerun instead of being quietly reused. We once had an env var point our verifier at a model deployment that didn't exist, and the runtime silently fell back to a different model. Nothing crashed, nothing logged red, the behavior just changed. Now a startup check refuses to run instead. In a validated system, loud failure is the cure.

No source, no action. Every extracted value carries an anchor: document version, section, exact quote, page, and the box on the page. Code verifies the quote really exists in the section the model was shown, and that every number appears inside it. Anything that fails goes to a human. Not dropped. Not quietly fixed.

For what the model missed, code finds the stretches of protocol that nothing covered, and a second model from a different family reads only that leftover text, hunting for omissions. Stopping rules. Dose tables. The footnote that rewrites a visit schedule. Two copies of the same model agreeing proves nothing, so we don't allow it.

And the single highest stakes check uses no model at all. Whether a criterion is inclusion or exclusion, the flip that would enroll exactly the wrong patients, is decided by plain code. Plain code cannot hallucinate.

History cannot be rewritten. Our database role can insert audit rows but cannot update or delete them. The application physically lacks the privilege. Each entry seals the previous one with a hash. Facts are never edited; they accumulate revisions, old value and new, with a name attached.

The payoff: from any value in a finished study build, you can walk back to the exact prompt and the exact raw model response that produced it. An auditor asking where a number came from gets a lookup, not an investigation.

We earned this one too. Reviewer decisions used to live in a JSON file we overwrote in place, so every save erased the history of the one before it. I still think about that file. It's all versioned database records now.

A human signs. Every time. Here's where the Elsa story comes back. Our reviewers check the model's work, same as FDA reviewers checked Elsa's. The difference is what the checking costs. Each item arrives with its evidence pinned to the screen, the original page, zoomed to the exact highlighted sentence. Verifying a value takes a glance, not a hunt. Clean items move fast. Flagged ones cannot hide in a bulk approve.

The reviewer's edits become the record. What ships is what the human decided, not what the model drafted, and approval is a Part 11 electronic signature: type your password again, state what the signature means, own it. The one step everyone expects us to automate, pushing a study live, we deliberately left manual. That's a person's call, on the record. I'd make that trade every time.

Five questions for any vendor, including us

Skip the demo. Every demo works. Ask these and watch how fast the answers come:

1. Where did this value come from? The sentence in the source document. Not a summary of it.
2. What checked it? Named steps, with versions, and what happens on failure.
3. Who approved it? The signature, its meaning, and what the reviewer changed.
4. Can you run it again? Same document, same result, byte for byte, years later.
5. What did it miss? And how would you know? Show the completeness accounting.

A raw model answers none of these. A well built system answers all five, possibly with the exact same model inside. The model was never the differentiator. The wrapper is.

One more thing. If a vendor leads with an accuracy percentage, ask "accuracy of what?" We refuse to publish one, because the claims worth making are the ones an auditor can test: every value traceable, every check versioned, every decision attributed, every run reproducible.

And I don't think building this way will be unusual for long. Look at where everything is pointing: the FDA's credibility framework, the joint AI principles FDA and EMA released this January, vendors quietly rebuilding around provenance.[11] The era of pointing a chatbot at a trial document is already ending. What replaces it is AI that shows its work. Grounded, checked, recorded, signed. The teams that build that way now will have nothing to rewrite when the final guidance lands.

Our platform is live in a paid Phase I trial today, and this layer is what my team and I work on every single day. In most industries the audit trail is overhead. In ours it's the point. The audit trail isn't the paperwork around the product. It is the product.

AccuraTrials builds the intelligence layer clinical trials run on. My co-founder Bhuwan wrote about why the industry was right to be skeptical in the first place: AI in Clinical Trials: Show Your Work, or Stay Out.