Can We Send PHI to an LLM?
Potentially, but a signed business associate agreement (BAA) is only one part of the decision. A healthcare team should send protected health information (PHI) to an LLM only after it understands the complete data path, has reviewed the provider's actual terms and configuration, has implemented appropriate application controls, and can show how the AI behaves when the evidence is missing or hostile.
The useful question is not simply whether an LLM vendor supports healthcare. It is whether this particular workflow can control PHI from collection through prompts, retrieved context, model outputs, tools, logs, support, deletion, and backups. That review should involve engineering, security, privacy, and qualified legal counsel.
Start with the PHI path, not the model catalog
HHS guidance treats risk analysis as an accurate and thorough assessment of potential risks and vulnerabilities to the confidentiality, integrity, and availability of electronic PHI. That scope follows the information wherever the organization creates, receives, maintains, or transmits it. A model choice made before mapping that path can hide important decisions behind a vendor name.
Draw the workflow from the first source record to the final user action. Include intake forms, EHR or partner integrations, file uploads, prompt assembly, retrieved context, model inputs and outputs, tool calls, vector indexes, traces, analytics, support tickets, exports, queues, caches, and backups. Then add the unhappy paths: a timeout, retry, exception, human escalation, support investigation, deletion request, and vendor termination.
The prompt is only one stop
Review the live request and the copies it can leave behind.
- 01 / Application
Source + authorized context
Choose the necessary fields and enforce access before retrieval.
Potential persistenceSource stores + indexInclude documents, chunks, metadata, and backups in the review. - 02 / Application
Prompt assembly
Combine the request with the context the user is allowed to access.
Potential persistenceTraces + queues + cachesInspect what is captured, who can read it, and when it is removed. - 03 / Provider boundary
Model input + output
Review the actual service, terms, and configuration in use.
Potential persistenceProvider-held copiesVerify retention, support access, and subprocessors for that service. - 04 / Application
Response or tool action
Check the output; enforce authorization again before a tool acts.
Potential persistenceHistory + exports + supportFollow the result into downstream records and recovery paths.
The HHS risk analysis guidance emphasizes identifying where ePHI is stored, received, maintained, or transmitted and documenting threats and vulnerabilities. For an LLM workflow, that means the prompt and the response deserve the same attention as the database record.
A BAA does not secure an architecture by itself
HHS explains that a cloud service provider can be a business associate when it creates, receives, maintains, or transmits ePHI on behalf of a covered entity or business associate. That can be true even when the provider stores encrypted information without the decryption key. The HHS cloud computing guidance also makes clear that a BAA must be paired with broader compliance work and risk analysis. Review the HHS business associate guidance alongside the provider's actual agreement and service configuration.
For an LLM provider, record the answer to each of these questions before approval:
| Review area | Question to verify | Evidence to keep |
|---|---|---|
| Agreement | Does the BAA cover the service and the actual use of PHI? | Executed agreement, scope, permitted uses, and contacts. |
| Retention | How long are inputs, outputs, abuse-monitoring records, and support copies kept? | Current terms, configuration, deletion process, and exceptions. |
| Training and reuse | Can prompts or outputs be used for training, product improvement, or another purpose? | Provider policy, account settings, and contractual restrictions. |
| Access | Who can access content, metadata, logs, or support diagnostics? | Roles, authentication, support path, and audit evidence. |
| Location and subprocessors | Where is information processed, and which downstream providers touch it? | Region, subprocessor list, data-flow map, and change notices. |
| Exit | How are primary records, indexes, traces, caches, and backups deleted or returned? | Termination procedure, retention exceptions, and verification record. |
A provider's marketing page is useful for discovery, but it is not the evidence packet. Confirm the service, plan, region, settings, and contract that your application will actually use. Do not infer that one product, endpoint, or account configuration applies to another.
Control the application boundary around the vendor
Even when a provider relationship is supportable, the application still determines what it sends and who can act on the result. Start with the minimum-necessary path. Remove fields that the task does not need, separate tenant context on the server, enforce authorization before retrieval and tool calls, and keep secrets out of prompts and logs.
Logging needs an explicit data policy. A trace that records the full prompt and output can become a second PHI store. Decide which events are necessary for auditability, which fields should be redacted or tokenized, who can inspect them, and how long they persist. Apply the same thinking to error reports, replay tools, analytics, screenshots, queues, local development environments, and backups.
The HIPAA Security Rule summary describes safeguards around the confidentiality, integrity, and availability of ePHI and calls for ongoing review of the effectiveness of security measures. In practice, access reviews, recovery tests, deletion checks, and change records are part of the boundary, not paperwork added after the model works.
Test model behavior before real PHI
Consider a fictional application that drafts a visit summary for an authorized reviewer. The team has reviewed its model provider, but an exception handler sends the complete request body to a separate error-monitoring service. The normal response looks correct; the failure path creates an additional data destination.
The unexpected copy.
A timeout can send visit text somewhere the normal request review missed.
- 01 / ApplicationSynthetic visitSummary request
- 02 / Model providerTimeoutNo summary returned
- 03 / Error serviceRequest copiedNew destination to review
Before / excessive capture
- event
- provider_timeout
- request_body
- Synthetic visit text
- destination
- Error-monitoring service
The handler copies the body even when the application log omits it.
After / proposed minimization
- event
- provider_timeout
- request_body
- Excluded
- destination
- Error-monitoring service
Keep reviewed diagnostic fields; verify SDKs do not add the body back.
What evidence should the review retain?
Simulate the timeout, inspect the exception event, retry queue, and support notification. Keep the sanitized sample, test result, destination inventory, and a named configuration owner.
This is a fictional failure scenario, not a customer incident or a compliance determination. It illustrates why the HHS cloud guidance calls for understanding the cloud environment and assessing risks across the ePHI a regulated entity handles. The appropriate contractual and legal conclusions depend on the actual workflow.
Infrastructure tests cannot tell you whether retrieved text contains an instruction that tries to redirect the model, whether one tenant's context appears in another tenant's answer, or whether a tool can perform an action beyond the user's authorization. Test those behaviors with synthetic or properly de-identified data before introducing real PHI.
Build cases for direct and indirect prompt injection, sensitive-output leakage, cross-tenant retrieval, malformed files, conflicting sources, missing evidence, overbroad tool permissions, retries, partial outages, and human escalation. Define the expected behavior before running the test: refuse, ask for clarification, cite the source, route to a reviewer, or stop without taking an external action.
The NIST AI Risk Management Framework is a voluntary resource for organizing trustworthiness work across design, development, deployment, use, and evaluation. It does not replace HIPAA analysis or legal review, but it gives a practical structure for turning model behavior into testable evidence.
Assemble evidence before authorizing production use
Readiness is easier to review when a team can point to a small, versioned packet. Keep the system boundary and PHI-flow map, risk analysis, vendor and subprocessor inventory, BAA and service terms, access matrix, retention and deletion design, backup and recovery evidence, AI evaluation results, incident runbook, named owners, training record, and change history together.
That packet should also show what remains unresolved. A pending contract question, an untested tool, or an unverified deletion path is not made safe by a confident launch date. Track the exception, owner, compensating control, and review date. Ask qualified privacy and legal counsel to evaluate conclusions that depend on the organization's role, contracts, or applicable law.
Evidence, with someone accountable
A signed contract is one artifact. Keep the operating evidence and unresolved questions together.
- Data path
- System boundary, data inventory, risk analysis, and the copies created along the request path.
- Vendor terms
- Applicable BAA, service configuration, subprocessors, and contract questions for qualified review.
- Access
- Roles, tenant boundaries, tool permissions, and evidence that restrictions are enforced.
- Retention + recovery
- Deletion design, backup behavior, recovery evidence, and unverified paths.
- Behavior
- Evaluation criteria and results for leakage, injection, missing evidence, and human escalation.
- Operations
- Incident runbook, named owners, training records, and change history.
Questions to ask before approving an LLM workflow
Can we remove PHI from the task?
Try synthetic inputs first. If real data is only redacted or minimized, treat that as exposure reduction, not proof of HIPAA de-identification. If data is de-identified, document the method and residual re-identification risk for the actual context. HHS recognizes Safe Harbor and Expert Determination as de-identification methods.
What happens to a prompt after the answer is returned?
Trace provider retention, abuse monitoring, support access, backups, indexes, caches, and deletion. The answer should describe the exact service and configuration, not a generic statement about AI.
Can a user make the model act beyond their authorization?
Authorization should be enforced by application services and tools, not delegated to the model's instruction-following. Test both the allowed and disallowed paths and keep evidence of the result.
What happens when the model has no reliable answer?
Define abstention, human review, uncertainty handling, and correction. A fluent response is not evidence, and a BAA does not replace an operational safety path.
Who owns the workflow after launch?
Name the people responsible for vendor changes, access review, model and data evaluation, incidents, retention, and release approval. Revisit the analysis when the model, prompt, vendor, region, data, or tools change.
Frequently Asked Questions
Does a BAA mean we can send any PHI to an LLM?
No. A BAA is an important contractual boundary, but the use case still needs a documented data path, risk analysis, application controls, vendor review, behavior testing, and operating ownership. Qualified privacy and legal counsel should review close interpretations.
Is an LLM automatically prohibited for a HIPAA workflow?
There is no useful yes-or-no answer without the facts. The decision depends on the organization's role, the provider relationship and terms, the information involved, the controls around the application, and how the workflow is operated.
Should prompts and outputs be stored for debugging?
Only when the purpose, access, retention, and deletion path are understood and support the risk decision. Full prompt and output capture can create additional sensitive-data stores that need their own controls.
Can we rely on de-identification for development?
Use synthetic data when possible. If data is de-identified, document the method and assess the residual re-identification risk for the context. A label without evidence is not a test strategy.
What can Software Sushi help with?
Software Sushi can help engineer the LLM workflow, including retrieval, server-side permissions, evaluation, logging boundaries, and operational handoff. An AI/ML consulting assessment can turn open risks into a prioritized implementation plan that supports a client's HIPAA compliance obligations; it is not a legal opinion or certification.
Ready to Review the Full PHI and LLM Path?
The right approval question is not whether an LLM sounds capable. It is whether the organization can explain and control the information, people, provider, model behavior, and operations around this workflow.
Start with our PHI readiness framework, then explore an AI/ML consulting assessment or book a discovery call.
This article is educational and not legal advice. HIPAA obligations depend on the facts, role, contracts, and applicable law. Have qualified privacy and legal counsel review close interpretations before production use.