Engineering project
Self-hosted PII/PHI redaction pipelines
Privacy engineering
Clinical notes, support conversations, and application records often mix useful context with information that downstream tools should not receive. Redactor separates detection, policy, transformation, and delivery so teams can decide what leaves a text-processing pipeline—and inspect how that decision was made.
Project status
Engineering project · v1 in development. Software Sushi has implemented configurable processing pipelines, an inspection interface, and offset-only decision evidence. This is a self-hosted development project, not a completed client rollout; production qualification and representative evaluation remain ahead.
The buyer problem: useful text, unnecessary exposure
Removing an entire record can destroy its value for analytics or AI. Passing it through unchanged can expose identifying details that the receiving application does not need. The practical challenge is to preserve appropriate context while applying a policy that operators can understand and test.
That policy has to answer concrete questions. Which fields must always be protected? Which detector may receive the original text? What does a review flag actually do? And can an operator explain a processing decision without opening another archive of sensitive documents?
Redactor addresses these as separate engineering concerns rather than treating a detector's confidence score as the whole privacy control.
Implemented capabilities
- Reusable pipelines. Accept items asynchronously, run configured detectors, resolve overlapping detections, apply a published policy, and produce an output payload with a metadata envelope.
- Configurable decisions. Ordered rules support masking, redaction, deterministic tokenization, blocking, flagging, allowing, numeric bucketing, and location generalization. The first matching rule wins.
- Coverage controls. Protect selected JSON field values independently of model recognition, or supplement Presidio with known sensitive terms under the configured policy.
- An inspection workflow. Try redaction, inspect output, configure a pipeline, and send synthetic text to its configured destination. Previewing a transformation and verifying delivery are distinct checks.
From submitted text to a delivery decision
- 1 · SubmitText + metadataAsynchronous acceptance
- 2 · DetectConfigured detectorsLocal or approved remote services
- 3 · Apply policyResolve matches and ordered rulesFirst matching rule wins
- 4 · TransformOutput + metadata envelope
- 5 · DeliverOne selected destinationFile / webhook / SQS / SNS
Read the boundaries: remote detectors can receive original text. Optional encrypted admission stores original input separately until decision-log commit. Delivery evidence is distinct from the decision ledger.
Three boundaries that shape the design
The evidence ledger is not a document archive
The decision ledger records positions and processing decisions without original document values. This gives operators evidence about policy application while keeping the ledger from becoming a second copy of the source text.
That exclusion applies to the ledger, not every store. Optional durable admission retains encrypted original input separately until the decision log commits. Sensitive-term lists live in configuration; destinations, backups, and database logs have their own retention boundaries. A useful privacy review maps each of these stores rather than relying on a blanket “no data stored” claim.
Self-hosted does not mean every detector is local
A selected local detector can process text within the chosen infrastructure boundary. Configured remote HTTP/gRPC detectors and AWS Comprehend Medical receive text at their respective services. Redactor's evidence reporting identifies configured external processors, making detector selection a data-flow decision as well as a coverage decision.
The operator therefore needs to approve the destination of original text before connecting a pipeline. Hosting the application and choosing where detection occurs are related, but different, controls.
A review flag is different from redaction
A flag leaves matched text unchanged and marks the item for review; it does not remove the matched value. Deterministic tokenization is not reversible and does not provide a token vault. These distinctions affect what downstream systems may safely assume about an output.
Field protection gives teams a deterministic option for selected keys in valid JSON objects or arrays. Known terms supplement detection but do not establish general language coverage. The bundled Presidio configuration is English. Core resolution, policy, transformation, and classification stages are pure functions, separating these decisions from runtime queues, storage, and destinations.
Implementation at a glance
Go processing service · PostgreSQL persisted state and decision evidence · React/TypeScript interface · Docker Compose local development · AWS Fargate deployment template.
The architecture keeps detector integration separate from policy and transformation. Teams can reason about a policy change without confusing it with a detector change, and inspect a processing decision without treating it as proof of downstream delivery.
Remaining work: qualify the complete data path
The next milestone is a reproducible evaluation using representative text, explicit coverage criteria, and destination checks. It should test missed identifiers and over-redaction, language assumptions, processing failures, access controls, and retention across input, configuration, evidence, output, and backups.
The current supported runtime is a single replica with stop-then-start updates. Default in-memory acceptance can lose unprocessed accepted items in a crash. Optional encrypted durable admission protects accepted input through decision-log commit, not downstream delivery; playground tests remain memory-only. Recovery and destination-failure behavior need to be qualified for the intended deployment.
For PHI workflows, appropriate organizational, legal, and security review also belongs in that qualification. Redaction alone does not establish HIPAA compliance or prove Safe Harbor de-identification.
Technical lesson: evaluate the boundary, not just the detector
A useful redaction control makes four things inspectable: what was detected, which policy applied, where the text traveled, and what delivery evidence exists. Separating those responsibilities is the project's central engineering contribution.
If your team is preparing sensitive text for analytics or AI, our custom AI software development and AI/ML consulting work can help define the data boundary and evaluation plan first.
Plan your text-processing boundary
Bring a representative workflow and the systems that need its output. We can start with the data flow, the protection policy, and the evidence needed to assess it. Discuss your data flow.
