Skip to content
Healthcare LLM development · RAG · PHI controls

Generative AI in healthcare that clinicians can check

For providers and health-tech teams: which generative AI tasks hold up in clinical settings, how to keep PHI inside a boundary covered by a business associate agreement, how to evaluate a model with your clinicians, and what the 2026 rule changes mean for what you build.

Doctor in a white coat with a stethoscope holding a smartphone in both hands
At a glance
  • Seven use cases, with limitsWhat each does well, where it fails, who signs off
  • PHI boundary by designBAA routes on AWS, Azure, Google Cloud and Anthropic
  • Evaluation clinicians ownGolden sets for accuracy, omissions, safety and bias
  • Fixed-scope proof of conceptOn synthetic or de-identified data, with a written report

How do you build generative AI for healthcare safely?

Start with a task where a clinician or trained staff member reviews every output, such as a referral summary or a draft reply to a patient. Keep PHI inside a boundary covered by a business associate agreement, send the model only the data the task needs, and retrieve from FHIR resources and approved documents with citations. Test against a clinician-labelled golden set for accuracy, omissions, safety and bias before go-live, log every prompt and output, and require a named person to sign off.

Generative AI in healthcare development

In healthcare, the hard part is not the model

A large language model can draft a discharge summary or answer a formulary question in seconds. Getting that into a clinic is a different job. The draft has to come from the right patient’s data and nothing else, cite where each statement came from, leave out nothing that matters (an allergy, an anticoagulant, a pending biopsy), reach a clinician who has time to check it, and leave a record of who approved what. Most failed healthcare AI pilots fail on those parts, not on the model.

We are a healthcare integration team first: FHIR, HL7 v2 and EHR interfaces are our daily work, and generative AI sits on top of that plumbing. This page sets out how we approach generative AI in healthcare development: which tasks we would build today and their honest limits, the reference architecture, where to host models under a BAA, how we evaluate with your clinicians, the US and Indian rules as of November 2026, and what drives cost per task. For general (non-clinical) assistants and agents see generative AI applications and AI agent development.

If you want to test an idea before committing a budget, we run a fixed-scope proof of concept on synthetic or de-identified data and hand you an evaluation report your clinical lead can read and challenge. We are not a medical device manufacturer and do not provide clinical or legal advice; where a feature edges towards diagnosis, we say so early and help you decide the regulatory route with your counsel.

Generative AI use cases in healthcare that hold up, and their limits

These are the tasks we would build today. Each keeps a person between the model and the patient, and each has a failure mode you must test for. The pattern is the same throughout: the model drafts, finds or summarises; a human decides.

Use caseWhat the model doesWhere it goes wrongHuman step
Healthcare RAG over policies, formularies and guidelinesAnswers staff questions from the approved version of an internal policy, a plan’s formulary or a clinical guideline, with a citation to the sectionSuperseded documents left in the index; formulary tiers that differ by plan and effective date; answering when the sources are silentStaff open the cited section before acting on anything clinical
Ambient documentation and draft notesTurns a recorded visit into a draft history, assessment and plan in your note templateMisheard drug names and doses, statements attributed to the wrong speaker, examination findings written in that never happenedThe clinician edits and signs every note; recording needs patient consent under your state’s rules
Patient-message and inbox draftingDrafts replies to portal or text messages from the thread and a short chart summary, and sorts messages by urgencyOffering new clinical advice in a routine reply; missing a red-flag symptom hidden in a long messageA nurse or clinician approves each clinical reply; red-flag messages go to a person, never to a draft
Prior-authorisation and appeal lettersAssembles a medical-necessity letter from chart evidence, matched to the payer’s published criteriaAsserting evidence the chart does not contain; citing an outdated policy versionStaff check each criterion against the chart; the clinician signs
Coding and CDI assistanceSuggests ICD-10-CM and CPT codes and documentation queries, each linked to the text that supports itCodes the documentation does not support, which is a compliance problem, not just an accuracy one; leading queriesCertified coders decide; queries follow your compliant query policy
Referral packet summariesReads a 30- to 200-page faxed or scanned packet and produces the reason for referral, history, medications, allergies and pending results, with page referencesPoor scans and handwriting; a critical item buried on page 87 left out of the summaryIntake staff review the summary with the page references open
Voice agents for schedulingBooks, moves and cancels appointments, answers practical questions and routes callsDrifting into clinical triage; disclosing details before the caller’s identity is verified; latency that makes callers hang upClear hand-off to staff, red-flag escalation, and callbacks for anything outside the script

Phone agents need their own engineering for latency, telephony and call transfer; our AI voice agents page covers it. Tasks we would not build as generative AI today: autonomous triage, dosing calculations, and anything that sends clinical content to a patient without a person reading it first.

Reference architecture: drawing the PHI boundary

Every component that sees protected health information sits inside one boundary, and every vendor inside it is covered by a business associate agreement (BAA) or kept away from PHI. In order of the data flow:

  1. Sources. The EHR through FHIR APIs (Patient, Condition, MedicationRequest, AllergyIntolerance, Observation, DocumentReference), HL7 v2 feeds, and document stores for policies and scanned packets. If your estate is mostly HL7 v2, a FHIR layer usually comes first; see HL7 to FHIR conversion.
  2. Minimum necessary. A task-specific extract: a reply draft needs the message thread, active problems, medications and allergies, not ten years of notes. Strip identifiers the model does not need, and replace those it does with tokens that are mapped back after the model call.
  3. Retrieval. Patient data is fetched live by resource ID, never stored in a shared vector index. Reference documents (policies, formularies, guidelines) go into an index with version, effective date, owner and status, and only current, approved versions are searchable. Permissions are applied before search, so a user only retrieves what they could open themselves.
  4. Model call. A BAA-covered endpoint in the right region, with training on your data excluded by contract, prompts versioned in source control, and output in a structured format that carries a source reference for each statement.
  5. Checks. Code verifies that every cited resource or page was actually in the context, flags statements without a source, and runs deterministic checks the model cannot be trusted with: allergy and medication cross-checks, dose units, dates.
  6. Review screen. The draft beside its sources, with unsupported statements highlighted. The reviewer edits, approves or rejects; nothing is sent or filed without that action.
  7. Write-back. Approved output returns to the EHR as a draft or preliminary document, with a FHIR Provenance record (or your EHR’s equivalent) noting that AI assisted and who signed.
  8. Audit log. Who asked, which patient, which sources, which model and prompt version, the raw output, the edits and the sign-off, kept under your retention policy. Tracing and observability tools see PHI too, so self-host them inside the boundary or cover them with a BAA.

On AWS, our HIPAA on AWS guide covers the account-level controls (encryption, logging, access reviews) this architecture assumes.

Where to host the model: BAA routes by vendor

What each vendor’s own documentation said on 7 November 2026. A BAA is necessary but not sufficient: you still configure the service correctly, and the BAA covers only the services and features it names.

RouteHow PHI coverage worksWatch for
Amazon BedrockListed, with Bedrock AgentCore, in AWS’s HIPAA Eligible Services Reference (updated 15 October 2026). PHI only after accepting the AWS BAA in AWS Artifact. For Claude on Bedrock, AWS is the data processor.Each supporting service (storage, search, logging, transcription) must also be on the eligible list. Anthropic’s separate “Claude Platform on AWS” offering is not HIPAA-ready.
Microsoft Azure (Foundry, Azure OpenAI)Microsoft’s HIPAA BAA is included by default through its Product Terms and Data Protection Addendum for in-scope Azure services (Microsoft Learn).Microsoft lists in-scope services, not individual models; confirm your deployment type is covered. Azure-sold models run abuse monitoring by default, and you can apply for modified monitoring. Claude in Microsoft Foundry needs a BAA with Anthropic plus a HIPAA configuration.
Google Cloud (Gemini Enterprise Agent Platform)Vertex AI was renamed Gemini Enterprise Agent Platform in April 2026. Google’s HIPAA page lists Generative AI on Agent Platform as covered under the Google Cloud BAA.Google says not to use pre-GA offerings with PHI unless their terms say otherwise, so check the launch stage of every feature you use.
Anthropic Claude APIHIPAA readiness with a signed BAA, enabled per organisation in the Claude Console with the standard BAA or through sales for a negotiated one (Claude API data retention).Once enabled it is permanent for that organisation, and non-eligible features are rejected with an error. The Batch and Files APIs are not eligible; prompt caching, citations and inline PDFs are. Keep PHI out of JSON schemas.

Our own products use the Claude API where they use a language model, with no fine-tuned models. We choose per client on data residency, existing cloud agreements and the eval results, not on habit.

De-identify where you can; minimise where you cannot

HIPAA recognises two de-identification methods: Safe Harbor, which removes 18 listed identifiers, and Expert Determination, where a qualified expert certifies that re-identification risk is very small (HHS guidance). Structured fields are easy to strip. Free text is not: names, dates, places and relatives turn up inside notes and scanned letters, so automated de-identification needs its own measured recall.

  • For a proof of concept, use synthetic data or a de-identified extract. Synthea, an open-source patient generator under the Apache 2.0 licence, produces FHIR R4, C-CDA and CSV records with realistic histories. Synthetic notes are thinner than real ones, so treat early scores as an upper bound.
  • For production, most clinical tasks need identified data: you cannot draft a reply to a patient without the patient. Then the controls are the BAA, minimum necessary extracts, tokenised identifiers where the model does not need them, and the audit log.
  • For evaluation sets built from real cases, de-identify and store them inside the boundary, with access limited to the people labelling them.

Evaluation: what we measure before a clinician sees a draft

Generic RAG metrics are covered in our RAG in production guide. Healthcare adds a severity scale and the measures below. Thresholds are set with your clinical lead before the first run, not after.

MeasureThe questionHow we score it
Critical omissionsDid the output leave out anything a clinician would need (allergy, anticoagulant, pending result, red-flag symptom)?Clinicians list must-include items per case; any miss is graded by severity
Unsupported statementsIs there anything in the draft that the sources do not say?Statement-level check against cited resources, automated first, then sampled by clinicians
Factual accuracyAre doses, dates, laterality and values copied correctly?Field-level comparison with the source record
Safety routingAre urgent messages and calls sent to a person rather than answered?Recall on a seeded set of red-flag cases; the target is agreed with your clinical lead
EquityDoes quality hold across age, sex, language, race and ethnicity where recorded, and payer?The same measures broken down by subgroup, with enough cases in each to mean something
UsefulnessDoes the draft save reviewers time?Edit distance between draft and signed version, and review minutes per item in the pilot

The golden set: typically 150 to 300 cases per task, chosen to include hard and rare ones, each labelled independently by two clinicians, with disagreements settled by a third. Every change to model, prompt, retrieval or template re-runs the set, and critical-severity errors block release. In the pilot, the same measures continue on a sample of live cases.

Human sign-off and the rules that require it

  • Drafts, not sends. Clinical content never leaves the system without a named person’s approval, and the approval is recorded.
  • Sources beside the draft. Reviewers should be able to check any sentence in one click. A review screen without sources produces rubber-stamping.
  • Patient disclosure. California’s AB 3030 has, since 1 January 2025, required a disclaimer and a way to reach a human on generative AI patient messages containing clinical information, unless a licensed provider reads and reviews them first. Texas requires practitioners to review AI outputs used for diagnosis (SB 1188, from 1 September 2025) and providers to disclose AI use to patients (HB 149, from 1 January 2026; see the Texas Medical Association summary). Check every state you serve.
  • Bias duties. The Section 1557 rule’s provision on patient care decision support tools (45 CFR 92.210) has required covered entities, since 1 May 2025, to make reasonable efforts to identify and mitigate discrimination risks in such tools. The equity breakdown above is your evidence.

Regulatory notes as of November 2026

United States

  • FDA clinical decision support. FDA revised its Clinical Decision Support Software guidance in January 2026. Software stays outside device regulation only if it meets all four criteria in section 520(o)(1)(E): it does not process medical images or signals, it displays or analyses medical information, it supports a health care professional’s recommendation, and the professional can independently review the basis so they do not rely primarily on it. The revision added enforcement discretion for software giving a single recommendation where clinically appropriate. It does not address patient-facing tools, and it contains no generative-AI-specific provisions.
  • ONC/ASTP decision support transparency. HTI-1 required certified health IT to support source attributes and risk management for predictive decision support interventions. The HTI-5 final rule, published on 28 October 2026 and effective from 28 December 2026, keeps the decision support criterion but removes those source-attribute and risk-management requirements. For builders, the transparency that matters now comes from your own documentation and your customers’ governance committees rather than from certification.
  • Prior authorisation. Under CMS-0057-F, impacted payers (Medicare Advantage, Medicaid and CHIP programmes and plans, and Exchange QHP issuers) must run a FHIR Prior Authorization API from 1 January 2027 (CMS fact sheet). Letter drafting still matters for appeals and for payers outside the rule.

India

  • DPDP Act and Rules. The Digital Personal Data Protection Rules were notified on 13 November 2025. The consent manager provisions apply from 13 November 2026 and most obligations on data fiduciaries, including notice, consent, security safeguards and breach reporting, from 13 May 2027. Health data has no separate “sensitive” category under the Act; it is personal data with the same duties. Our DPDP checklist sets out the duties.
  • Health AI guidance. ICMR’s 2023 Ethical Guidelines for the Application of AI in Biomedical Research and Healthcare remain the main ethics reference. In February 2026 the Ministry of Health launched SAHI, a national strategy for AI in healthcare, and BODH, a benchmarking platform for health AI built with the National Health Authority.
  • Medical device software. CDSCO finalised its guidance on medical device software on 30 July 2026. Software with a medical purpose, including AI-enabled software, is classed A to D under the Medical Devices Rules, 2017, and AI submissions must describe training data and performance across Indian sub-populations.

This is engineering guidance, not legal advice. Your counsel decides how these rules apply to your product.

What drives cost per task

Model tokens are rarely the largest line. Illustrative, not client work, at Anthropic’s list prices read on 7 November 2026 (Claude pricing): Claude Sonnet 5.5 at $2 per million input and $10 per million output tokens, Claude Haiku 5.5 at $0.10 and $0.50 for prompts up to 100,000 tokens.

TaskAssumed tokensModel cost per taskReviewer time per task
Referral packet summary28,000 in, 1,200 out on Sonnet 5.5, plus a 30,000-token citation check on Haiku 5.5About $0.0713–5 minutes of intake review
Prior-authorisation letter draft20,000 in, 1,500 out on Sonnet 5.5About $0.0555–10 minutes to check criteria and sign
Patient-message reply draft6,000 in, 300 out on Sonnet 5.5About $0.0151–2 minutes to approve or edit
Policy or formulary question8,000 in, 400 out, on Sonnet 5.5 or Haiku 5.5About $0.020, or $0.001 on HaikuReading the cited section

The drivers that move these numbers:

  • Context size. Whole charts cost more and often answer worse than a task-specific extract.
  • Data residency. US-only inference costs 1.1 times the standard price on Claude 4.6 and later models; cloud platforms set their own regional prices.
  • Discount eligibility. On the Claude API, prompt caching is HIPAA-eligible but the half-price Batch API is not, so PHI workloads there pay full rate for overnight jobs.
  • Speech. Ambient documentation and voice agents add transcription billed per audio minute, usually more than the tokens.
  • Review minutes. At clinician rates, two minutes of review outweighs every line above. The design goal is a draft that is quick to verify.
  • Evaluation upkeep. Re-running the golden set and labelling new failure cases is a monthly cost in clinician hours.

For build effort and a 12-month total for an agent, see our AI agent development cost guide.

The proof of concept, in detail

One use case, fixed scope, fixed price agreed after a scoping call. It runs in your cloud account, or ours if you have none yet, on synthetic data from Synthea or a de-identified extract you provide.

  • Week 1: the task, the output template, the must-include items and the release thresholds, agreed in writing with your clinical lead.
  • Weeks 1–2: data preparation and the golden set: 150 to 300 cases, labelled by your clinicians with our tooling.
  • Weeks 2–5: the working prototype with retrieval, citations, checks and a review screen, iterated against the golden set.
  • Final week: the evaluation report: scores by measure and subgroup, an error catalogue with examples, cost per task, the PHI boundary design for production, and a go or no-go recommendation.

Typical effort is 4–7 engineer-weeks for a document task (summaries, letters, RAG) and 6–10 for ambient documentation or a voice agent, confirmed after the scoping call. Plan for 10 to 20 hours of clinician time for labelling and review.

Questions to ask any AI in healthcare development company

  1. Which tasks would you refuse to build, and why?
  2. Where exactly is the PHI boundary, and which vendors inside it have signed a BAA?
  3. Who labels the golden set, how many cases, and what blocks a release?
  4. How do you measure omissions, not just errors?
  5. How does a reviewer check a sentence against its source?
  6. What is logged for each output, and for how long?
  7. How would you know if quality drifts after a model update?
  8. Who owns the prompts, evaluation set and code at the end?

When generative AI is the wrong tool

  • When rules work. Structured templates filled from discrete data are cheaper and easier to validate than a generated note. Our own OneHealth ambient-notes module drafts notes from rules, not a language model.
  • When the answer is a lookup. Eligibility, appointment slots and lab values come from systems of record through APIs, not from a model’s memory.
  • When nobody can review. A draft that saves a nurse three minutes but needs a doctor’s ten is a net loss.
  • When the output would be the decision. That is device territory, with a different budget and timeline.
Services

Healthcare generative AI services

Every engagement keeps PHI inside a boundary you control and ends with documents your clinical and compliance leads can review. Effort figures are typical and confirmed after scoping, priced as a fixed fee per phase.

Fixed-scope proof of concept

The problem

You have a promising use case, but no evidence yet that it is accurate enough, safe enough or cheap enough to justify a production budget.

How we do it

One task on synthetic or de-identified data: golden set with your clinicians, working prototype with citations and a review screen, then an evaluation report with scores by measure and subgroup, an error catalogue and cost per task. Typically 4–7 engineer-weeks for a document task, 6–10 for ambient or voice.

What you get

  • Golden set you keep
  • Working prototype in your account
  • Evaluation report with error catalogue
  • Production PHI boundary design
Pilot plan

How a proof of concept runs

Typical for one document task. Voice and ambient pilots add simulated-call or recorded-visit testing.

Week 1

Task and thresholds

Use case, template, must-include items and release thresholds agreed with your clinical lead; data source chosen (Synthea or a de-identified extract).

Weeks 1–2

Golden set

150 to 300 cases labelled independently by two clinicians, disagreements settled by a third, stored inside the boundary.

Weeks 2–5

Build and iterate

Retrieval, citations, deterministic checks and a review screen, re-scored against the golden set after every change.

Final week

Report and decision

Scores by measure and subgroup, error catalogue, cost per task, production boundary design and a go or no-go recommendation.

Senior engineers firstYour first call is with someone who will do the work, not a sales script.
You own everythingCode, configuration, reports and documentation stay in your accounts and are handed over.
We run our own products15 live platforms we build and operate, so we feel production pain too.
Honest about fitIf a cheaper route or another provider suits you better, we will say so.
FAQ

Questions about generative AI in healthcare

The tasks that hold up are ones where a person reviews every output: answering staff questions from policies, formularies and guidelines with citations; drafting visit notes from recorded conversations; drafting replies to patient messages; assembling prior-authorisation and appeal letters; suggesting codes with supporting text; summarising referral packets; and scheduling calls. Autonomous triage and unreviewed clinical messages to patients are not ready.

As of November 2026: Amazon Bedrock is on AWS’s HIPAA-eligible list under the AWS BAA; Microsoft includes a HIPAA BAA by default for in-scope Azure services; Google lists Generative AI on its Gemini Enterprise Agent Platform, formerly Vertex AI, under its Cloud BAA; and Anthropic offers HIPAA readiness for the Claude API with a signed BAA. Each covers only named services and features, so check yours.

Retrieval-augmented generation in a clinical setting: the system finds the relevant passages in approved policies, formularies or guidelines, or the relevant FHIR resources for a patient, and the model answers only from them with citations. The healthcare-specific work is keeping only current, approved versions searchable, applying permissions before search, and never mixing one patient’s data into another’s context.

It depends on intended use. Under FDA’s clinical decision support guidance, revised in January 2026, software aimed at health care professionals stays outside device regulation only if it meets four criteria, including that the professional can independently review the basis of its output. Drafting and summarising for review usually sits outside; software that analyses images or effectively makes the decision does not. Ask your regulatory counsel.

Against a golden set of typically 150 to 300 real or realistic cases per task, labelled independently by two clinicians. We measure critical omissions, unsupported statements, factual accuracy, safety routing of urgent cases, performance across patient subgroups and reviewer time saved. Thresholds are agreed with your clinical lead beforehand, and critical-severity errors block release.

Yes. Synthea, an open-source synthetic patient generator, produces FHIR R4, C-CDA and CSV records with realistic histories, and a de-identified extract is the other option. Synthetic notes are thinner than real ones, so scores from a synthetic pilot are an upper bound, and we re-test on real data inside your BAA-covered boundary before production.

Model tokens are usually cents. In our illustrative examples at Anthropic’s November 2026 list prices, a referral packet summary cost about 7 cents and a patient-message draft about 1.5 cents. Transcription for ambient notes or voice is billed per minute, and the largest cost is reviewer time, so we design drafts to be quick to verify.

The HTI-5 final rule, published on 28 October 2026 and effective from 28 December 2026, keeps the decision support interventions certification criterion but removes the source-attribute and intervention risk management requirements that HTI-1 added for predictive tools. Transparency now depends more on what developers document and what provider governance committees ask for.

Rarely. Most healthcare tasks are better served by retrieval from current sources, structured prompts and deterministic checks, which are easier to update and audit. Our own products use the Claude API without fine-tuned models. We consider fine-tuning only when a stable, well-labelled task needs consistent format or classification that prompting cannot reach.

Last updated