Building an AI audit trail an examiner will accept
The hard question is what the system was set to do on a given day. Here's how to answer it, from a log of every settings change to exported evidence.
Read about audit trailsHealthcare · 4 min read
Healthcare AI projects rarely stall on the model. They stall in privacy review, on four engineering questions that can each become a setting you can point at.
Most healthcare AI projects that stall do not stall because the model was not good enough. They stall in privacy and legal review, where the questions are engineering questions and the project has no engineering answers to them. This is a walkthrough of the four that come up almost every time, and of how each one becomes a setting somebody can point at, test and defend, instead of an assurance somebody gives in a meeting.
Scope note: this is engineering guidance about controls, not legal advice or a statement of what any regulation requires. NeuralSeek holds SOC 1 and SOC 2 Type II reports and is HIPAA compliant. Your counsel, privacy office and internal audit decide what applies to your organization.
The common assumption is that the regulation forbids putting clinical data anywhere near a language model, and so the project waits for a legal answer that never quite comes. In practice the review is rarely asking for permission in the abstract. It is asking four concrete things: what reaches the model, what record exists afterwards, what the system is able to see at all, and how any of that can be demonstrated.
Each of those is answerable in configuration, and a project that arrives with the answers configured has a very different review from one that arrives with a demonstration and a promise.
The strongest possible answer to this question is 'none of it', and it is achievable more often than teams expect. If protected information is removed before the prompt is assembled, the question of what the model provider does with it stops being interesting.
The strongest setups redact in two passes for the reasons set out in the article on PII detection. A fast deterministic pass removes the structured identifiers (record numbers, dates, contact details) at negligible latency. A contextual pass then catches what patterns cannot: a name in a free-text clinical note, a location phrased a dozen ways, a description that identifies a person without naming them.
The third decision is what happens on a match, and it is worth setting deliberately per workflow: mask the value, substitute a stable token so the surrounding sentence still makes sense, remove it, or refuse the request. Set once, applied consistently, and visible in the configuration rather than in someone's code.
You cannot account for access you did not record, and a review will assume nothing about what your system did. Two complementary records do most of the work: a durable, tamper-evident organization-wide log of every interaction, and a record of the prompts themselves so that a question raised months later can be reconstructed with the context the system was actually working from.
The practical test is whether producing that record is an export or a project. If a reviewer asks what a specific assistant said to a specific person on a specific day and the answer takes a week to assemble, the control does not really exist yet.
Redaction and logging matter little if a retrieval can reach material it should never have been able to reach. This is an access question, not just a login question: an assistant built for one department should be structurally incapable of retrieving another department's records, not merely discouraged from doing so.
That is enforced at the retrieval layer, by constraining which documents each tenant or business unit can query, before any ranking happens. Alongside it, credentials and connection secrets should be kept out of prompts and responses entirely. Otherwise the evidence file you produce for a reviewer becomes a place a key can leak from.
The last question is the one that decides how long the review takes. A configuration that is correct but undocumented gets treated as an assertion. The organizations that clear review quickly are the ones that can put each control next to the requirement it is there to satisfy, in a structure the reviewer already knows.
That mapping is the difference between a security questionnaire that becomes a months-long evidence hunt and one that is answered from a document. It is also, usefully, the record that survives staff turnover.
When NeuralSeek runs inside your environment, the compliance scope is yours and the data never has to leave it. The controls above are ours to provide and yours to configure and operate.
NeuralSeek is audited for SOC 1 and SOC 2 Type II and is HIPAA compliant.
Keep reading
Practical reading on building, governing and running AI agents.
For settings and parameters, see the product documentation.
The hard question is what the system was set to do on a given day. Here's how to answer it, from a log of every settings change to exported evidence.
Read about audit trailsWhy the model can't fix it alone, what an attack hidden in a document looks like, and how to contain it.
Read about prompt injectionSign up online and start with a two-week trial, or talk to a NeuralSeek expert about running it in your own cloud.