Case Study - Deciding Where AI Supports a Clinician — and Where It Doesn’t
Clinical decision support for clinicians, education for members, and no unsupervised AI judgment in between — a deliberate line drawn feature by feature, and built into the architecture rather than left to policy.
- Client
- Metabolic Health Platform
- Year
- Service
- Clinical Decision Support & Regulatory Design
- My role
- Fractional CTO
Every AI feature in the platform was decided on purpose: lean into decision support where a clinician can see and check the basis, gate anything that reaches a patient behind a physician, and keep AI out entirely where it would become the decision-maker.
The challenge
The platform turns a member's health survey and lab results into an individualized assessment, and gives clinicians an AI they can question about any member's chart. Each AI feature raised the same question in a different form: is this software helping a clinician decide, or deciding for them? The answer shapes how a feature must be built, what it is allowed to say, and how it is regulated — and it's far cheaper to settle at design time than after launch.
My role
I set the clinical decision support posture for the product — which features would lean in, which would be gated, and which would stay out of AI entirely — and carried it through into the requirements, the architecture and the prompts. I worked across the engineering on the clinician workspace, the member app and the AI pipeline, and with the clinicians who use and review its output.
Leadership decisions
Decision 1
Lean into lab interpretation — but take it away from the model.
Instead of: Letting the LLM read the labs and assign each result a tier, which is how the first version worked.
Early versions let the model tier lab results, and it assigned some incorrectly. Lab interpretation is exactly the kind of support clinicians want, so rather than drop it, I moved it into a deterministic rules engine over a versioned reference library, with explicit decision order: an under-18 member short-circuits everything, a unit error or missing cohort returns "interpretation unavailable" rather than "looks fine," and a lab-flagged critical result outranks any band. The model now receives results already tiered and is told not to re-tier them. A clinician-readable guide is generated by running the real engine, so the documented logic can never drift from the code.
Decision 2
Let the AI write for members — only behind a physician, and only as education.
Instead of: Publishing AI-written assessments straight to members once generation succeeded.
The member assessment is the highest-stakes AI output, because it reaches real patients. So a physician reviews, edits and publishes every one, and the member can only ever read the published version. The language is held to education: never diagnose ("a pattern consistent with," not "you have"), never prescribe ("worth discussing with your provider," not "you should"), no trends from a single draw. Disclaimers and crisis resources come from the platform's template, never from the model, so they can't be dropped or reworded. After publishing, members are offered a review visit with their provider.
Decision 3
Keep AI out wherever it would become the decision-maker.
Instead of: Building every AI feature that was technically possible.
Members have no AI chat — every AI conversation is clinician-side. Critical lab values come from the performing lab's own flags and go to a clinician queue; the platform never invents its own thresholds. Ordering and dosing remain clinical acts with no AI logic behind them. And an AI feature that analyzed images of lab reports — the closest thing in the product to device-like analysis — was taken out of the product.
The solution
A clear posture, carried all the way into the system: decision support for clinicians, education for members, and no unsupervised AI judgment in between. Where it matters most, the posture lives in the architecture — a rules engine that owns interpretation, a publish gate the member app can't get around, and an AI chat that only clinicians can open.
That lines up with the four criteria US law uses to separate clinical decision support software from a regulated medical device:
| The criterion | How the design lines up |
|---|---|
| Doesn't analyze a medical image or device signal | Inputs are discrete lab values and survey answers; the image-analysis feature was removed |
| Displays or analyzes medical information | Lab results, intake answers and chart data |
| Supports a clinician rather than replacing one | AI chat is clinician-only; every assessment needs a physician to publish it |
| The clinician can independently review the basis | A deterministic engine with a generated, clinician-readable guide; the AI's original and every edit are kept |
The one place those criteria don't settle on their own is the member-facing assessment, which is why it carries both the education-only language and the physician gate. This was a deliberate design posture, built into the product — not a substitute for a formal regulatory determination, which belongs with regulatory counsel.
Lab interpretation is governed, not generated — the one place the model doesn't get to be creative.
Impact
- Decision support clinicians can check. Every lab tier comes from rules a clinician can read, with a guide that is generated from the engine itself.
- No AI output reaches a member unreviewed. A physician stands between the model and every patient, by construction.
- The product's regulatory posture was decided early, on purpose. Each AI feature has a stated reason for sitting where it does — including the ones that were deliberately not built.
Under the hood — the technical detail
The rules engine is pure JavaScript over a versioned lab reference library — 81 markers across 94 sex- and age-specific cohorts and 337 bands, many using the performing lab's own limits as boundaries. Derived markers such as HOMA-IR, FIB-4, the triglyceride-to-HDL ratio and the neutrophil-to-lymphocyte ratio are computed and then evaluated like any other result. A build script imports and runs the real engine to produce the clinician-readable interpretation guide, with worked examples.
The assessment pipeline waits for both intake and evaluated labs, then fans out four section-generation jobs, each a single Claude call with extended thinking, a private analysis pass and a self-check before output. The prompts carry the language rules; the assembler template carries the disclaimers. Clinician chat is grounded in the member's full chart, assembled by code rather than looked up by the model, so every answer rests on the same reviewable view the clinician has.
Every AI step is a code-orchestrated workflow with one narrow job, not an autonomous agent: no model decides what runs next. That keeps each step predictable, auditable, and individually retryable.
- Clinical decision support design
- Deterministic clinical rules engines
- Human-in-the-loop LLM review
- Patient-facing AI language safety
- FDA non-device CDS criteria