Stopping an LLM fabricating clinical facts across 2M+ documents

The client: a healthcare SaaS platform processing high volumes of clinical documentation.

The problem

An LLM summarization pipeline was processing clinical documents at scale — well into the millions — extracting structured information for downstream use. In that setting, a plausible-sounding wrong answer is a materially worse failure than an obvious one: the summary has to be not just fluent, but constrained to what the source document actually supports.

Approach

The pipeline pairs prompt engineering with hard constraints rather than relying on prompting alone: extractive rather than generative summarization where possible, so the model is constrained to content that exists in the source; structured outputs validated against a schema before anything downstream trusts them; and event-driven processing so failures are caught and retried rather than silently dropped.

Underneath that sits a monitoring and evaluation layer — not a one-time test suite, but a running check on production output, because a pipeline processing this volume will drift, and the point of the guardrails is to catch that drift before it reaches a downstream consumer, not just to look good in a demo.

Outcome

The pipeline has processed over 2 million documents in production, with safety guardrails and evaluation running continuously rather than as a pre-launch checklist item.

What I’d tell you if you were building this

Extractive constraints buy you more real safety than a longer, more carefully worded prompt does. If the output can be checked against the source, check it — don’t just ask the model to be careful.