Case Study - The Data Model That Makes an Unreviewed AI Report Unreachable

Overview

The client delivers personalized care plans built from a member's health survey and blood work. Every member needs an individualized clinical assessment — writing one by hand for every member doesn't scale past a small caseload, so the obvious move is to let an LLM read the survey and labs and draft the narrative.

That obvious move is also the dangerous one. Two categories of decision in this pipeline are clinically load-bearing: which treatment pathway a member is on, derived from a substance-use screening question, and whether a member has enough data — survey and labs — to be assessed at all. Getting either wrong isn't a copy problem; it's a wrong-report-reaches-a-patient problem.

The problem

We needed to auto-draft clinical narrative at scale while making sure the LLM never got to make the decisions that determine what actually gets said. And once a draft existed, we needed a second guarantee that was, if anything, harder to build: the assessment-generation system, the clinician-facing EMR, and the patient-facing app are three separately deployed repositories on two separate Cognito user pools. None of them trusted each other's APIs — deliberately, since a compromised or buggy cross-pool call is a direct path to a patient reading another patient's data, or reading a draft nobody had reviewed yet.

Why it took a physician-engineer

The substance-use pathway classifier has real clinical meaning — it determines which of two structurally different assessment paths a member's report follows. Encoding that as a deterministic function, rather than an LLM judgment call, required understanding why the distinction matters clinically, not just that it exists. A generalist engineering team would more likely have let the model infer pathway from context, or generated a partial report the moment survey data arrived.

The publish workflow's most consequential design choice — preserving the LLM's original output the moment a clinician makes their first edit, then snapshotting every subsequent edit — isn't a generic audit-log pattern borrowed from software engineering. It's there because a clinician's edit to a generated clinical report needs to be reconstructable: what did the model say, what did the physician change, and when. A generalist engineer building "a CMS for editing reports" reaches for version history as a feature. Understanding that this is specifically about preserving the provenance of a model-generated clinical claim versus a physician's clinical judgment is what shaped it instead into a permanent original-vs-edited split with immutable snapshots.

Approach

Nothing clinically important is decided inside a model call. Intake gets normalized to stable internal keys before anything downstream touches it. Pathway is derived by a pure function, run once, and handed to generation rather than re-derived by the model. The four sections of the report generate independently and in parallel, and every generation produces both an HTML narrative a clinician can edit and a structured JSON bundle from a single pass. Publishing freezes both artifacts, so a later edit can't silently change what a member already saw. No report reaches a member without a clinician reviewing — and optionally editing — first.

The organizing decision on the publish side was architectural: the database is the integration point, not the API. Rather than the EMR calling an assessment-infra API, or the patient app calling the EMR, each of the three systems reads and writes the same shared DynamoDB tables and S3 bucket directly, through its own Lambdas, authorized by its own Cognito pool. This sidesteps the two-Cognito-pool trust problem entirely, and it keeps each repo's blast radius contained: the patient-app repo's Lambda physically cannot serve a working draft, because the field it reads from doesn't exist until a clinician publishes.

The member-facing read endpoint only ever fetches a publishedReportS3Key field that's null until the publish step sets it. That isn't an application-level check that a report is "published" before showing it — the data doesn't exist yet from the patient app's point of view until it's published.

Under the hood

A DynamoDB Streams handler normalizes raw intake answers to internal keys and writes them to the patient's record; a second stream handler fires only once both normalized intake and a completed lab evaluation exist, atomically claims an assessment ID, and enqueues to SQS. The consumer runs the canonical pathway classifier, creates an assessment shell, and fans out four section-generation jobs to a second queue, each calling Claude via the Anthropic SDK, streamed. An assembler waits for all four, builds both the HTML and JSON report, and uploads to S3.

On the publish side, S3 holds the report at each stage of its lifecycle — canonical (mutable during review), .original.html (preserved on first edit), timestamped edit snapshots, and .published.html (the only thing the member app can read). State transitions are enforced with DynamoDB conditional writes rather than application logic, so a race between two clinicians publishing simultaneously can't corrupt state — it can only over-increment a version counter, which is explicitly tolerated. Every log line in the pipeline follows a PHI-safe convention: no answer content, no lab values, no report text — only IDs, statuses, and S3 keys.

Outcome

Real members now flow through the assessment pipeline with zero manual staff intervention on the data-ingress side — webhook-driven intake and labs replace what used to require an admin to upload data by hand. The structural guarantee at the center of the problem holds regardless of future changes elsewhere in the codebase: a patient cannot see a report before a physician has reviewed and published it, because the data enforces it, not a status check a future change could accidentally bypass.