Case Study - Automating the Read, Never the Judgment
An AI triage layer for adverse reaction reporting, scoped by a single question — what does a wrong answer cost in each direction — so the model does the reading and a human keeps the call.
- Client
- Primum
- Year
- Service
- AI Operations
We built an AI process to review adverse reaction reports: the model does the reading and the sorting, and a human makes every call that carries regulatory weight. The engineering was the easy part. The work that mattered was deciding which parts of the process could be automated at all.
Overview
Adverse reaction reports arrive as prose. They come from clinicians, patients, and intermediaries, in whatever form the reporter felt like writing, and someone has to read every one to decide whether it describes a reportable event, how serious it is, and how fast the clock is running.
The reading is repetitive. The decision is not. That gap is where the opportunity was.
The problem worth solving wasn't the one we were asked about
The instinct with a queue of unstructured text is to ask whether an LLM can classify it. It can — that part is a weekend of work now. The question that determines whether the project survives contact with reality is different: what happens when it's wrong, and in which direction?
For adverse reaction reporting the two failure modes are not remotely equivalent:
- A false negative — the system reads a real reportable reaction and concludes it isn't one — is a regulatory failure that surfaces months later, if it surfaces at all. Nobody audits the reports that were never opened.
- A false positive — the system flags something that turns out to be routine — costs a reviewer a few minutes.
Once you write those two sentences down, the architecture stops being a matter of taste. The model may never conclude "not reportable." It is allowed to narrow, rank, extract, and summarize. It is not allowed to close anything.
The scarce skill in AI operations isn't wiring a model to an inbox. It's knowing which part of a process is safely automatable, and which part will quietly generate errors nobody catches for three months.
Why the ops knowledge had to come first
You cannot make this call from outside the process. Deciding what an AI may safely do to adverse reaction reporting requires knowing what the reviewer is actually doing when they read one — which signals in a narrative make something serious rather than merely unpleasant, why an expedited timeline starts when it does, what a regulator expects to see reconstructed if they ask a year later, and which ambiguities a human resolves so instinctively they never appear in any written procedure.
That last category is the one that kills pilots. The undocumented judgment is invisible in a process map and obvious in a transcript. It is why an automation designed from a flowchart looks correct and performs badly, and why the assessment has to be done by someone who can read both the clinical content and the system.
Approach
The design followed the error asymmetry rather than the capability:
The model reads; it does not decide. Every report still reaches a human. What changed is what the human receives: a report that arrives pre-read, with the reaction described, the candidate seriousness signals extracted and quoted from the source text, and a suggested priority.
Extraction is grounded in the source. Every extracted field points back at the span of narrative it came from. A reviewer verifies by looking, not by re-reading the whole thing and trusting a summary — and a summary that can't be traced to its source is not reviewable, it's just faster to be wrong with.
Uncertainty routes up, never down. Low model confidence escalates priority. The system's failure mode under ambiguity is to waste a reviewer's attention, which is a cost that shows up immediately on someone's desk rather than silently in a compliance file.
The trail is the deliverable. What the model saw, what it produced, which version produced it, and what the human did with it. A process that can't reconstruct its own reasoning is not defensible under audit, regardless of how well it performs day to day.
What generalizes
The pattern transfers to most healthcare ops work, and the sequence matters more than the tooling:
- Inventory the process as it is actually performed, not as documented.
- Score each step on four axes: volume, cost of a wrong output, PHI exposure, and reversibility.
- Automate the high-volume, low-consequence, fully reversible reading. Leave the judgment.
- Instrument the boundary so the handoff between the two is visible and auditable.
Most of a process fails at least one of those tests. Prior authorization, eligibility checking, intake triage, referral routing, and chart abstraction each have a different error tolerance, and treating them as one category — "ops work an agent could do" — is the most common way these projects quietly fail.
The map is worth more than the build. Once a team can see which 20% is safely automatable, the automation is straightforward and the remaining 80% stops being a source of unpleasant surprises.