Case Study - Making an LLM Prove Every Claim Before It Reaches a Surgeon
An operative-note generation pipeline where a language model may propose a clinical fact, but a deterministic downstream check — independent of the model — decides whether it ever reaches "populated" status.
- Client
- Otto
- Year
- Service
- Regulated AI & LLM Safety Architecture
We turned a fabrication-prone language model into a documentation tool a surgeon can actually rely on, by making the model prove every claim against a verbatim transcript quote before it counts as fact.
Overview
Otto is a hands-free intraoperative documentation product: a surgeon dictates through the case, and the audio flows through transcription into a structured note draft, review, and approval. The core promise was a usable operative-note draft within minutes of case end. The core risk was the same technology that made that possible — a language model that can, by default, say things that aren't true.
The problem
The obvious approach — ask the model to be careful, prompt it not to hallucinate — is not a control. It's a hope. We needed a way to guarantee, structurally, that nothing reaches a surgeon's eyes as "populated" unless it traces back to something actually said, without relying on the model's self-report of its own reliability.
Why it took a physician-engineer
Deciding which fields are safe to ever leave ambiguous and which must never be inferred is a clinical judgment, not an engineering one — a wrong "left" instead of "right," or a fabricated implant ID, is categorically worse than an honest blank. A generalist engineer building this as a form-filling problem optimizes for filling in more fields, because that's what "the AI extracted more information" looks like in a demo. The right instinct here was the opposite: architect the system to prefer an honest "not stated" over a plausible-sounding guess, and make that preference enforceable in code, not just in a prompt.
Approach
The generation pipeline runs on a rule set from day one: the model only proposes; a deterministic step downstream verifies. The model is forced, via tool-use against a fixed schema, to emit exactly one entry per required note field, each with an explicit status of populated or not_stated, and — critically — an exact verbatim quote from the transcript for anything it populates.
Never trust a model's self-reported character offsets for where its evidence lives. Make it quote instead, and verify the quote independently.
A downstream Lambda then re-locates that quote in the raw transcript by exact string match. If the quote isn't found character-for-character, the field is downgraded to not_stated — no matter what the model claimed. That trades a small amount of generation flexibility (the model must produce exact substrings, not paraphrases) for a system that structurally cannot fabricate provenance, even if it fabricates content.
Under the hood
Generation calls the model via a direct API integration using forced tool-use against a JSON schema constrained to the procedure's required field keys — the schema itself is resolved per surgeon and per procedure at generation time, not a hardcoded constant, falling back to a stable default set when a surgeon hasn't customized a procedure yet. A separate Lambda performs the exact-match verification and, for anything that clears the gate, attributes the transcript span back to the specific captured audio segment it came from. The review screen surfaces every populated field next to its supporting transcript span; "not stated" fields are visually flagged rather than hidden, and no note carries finalized status until a human explicitly approves it through a single, dedicated code path. Every edit and the approval itself are written as append-only log entries, forming the audit trail. Orchestration runs as a Step Functions workflow: transcribe → domain post-correction → generate → verify provenance → write draft.
- Regulated AI
- Clinical product strategy
- Human-in-the-loop safety architecture
- LLM system design
- AWS serverless (Lambda, Step Functions)
Outcome
The pipeline has been verified end-to-end against synthetic orthopedic case dictations with real transcription and generation calls, including the approval-gate and edit-log paths — and the evidence gate has already caught and correctly downgraded fields where the model's proposed quote didn't literally match the transcript. A surgeon reviewing a draft is reviewing claims that are individually falsifiable against the transcript, not opaque model output.