Applied AI
Meddies-PII: Synthetic Data for Multilingual Clinical De-identification
The synthetic data pipeline behind a compact PII model for clinical text — 1M documents in 17 languages, filtered by 13 deterministic validation gates.
Key results
- synthetic clinical documents
- 1M
- 17 languages, 9 PII labels
- deterministic validation gates
- 13
- regex, character-offset parser, rule checks, SHA-256 de-duplication
- mean exact-match micro-F1
- 0.827
- downstream 350M model on 15 external benchmarks; strongest baseline (OpenAI Privacy Filter) 0.658
- F1 on adversarial benchmark
- 0.878
- 5,100 documents; baseline 0.540
Problem#
Clinical conversations are full of names, phone numbers, patient IDs and insurance numbers. Before any of that text can be summarised, analysed or used for training, those identifiers have to be found and masked — in every language the product supports.
Training a detector needs labelled data, and real patient records are exactly the data you cannot pass around freely. So the project depends on synthetic clinical documents that are realistic enough to learn from, with labels that are exactly right.
My role#
I designed and maintain the data-generation pipeline. The model itself was trained by teammates; I am co-first author of the paper describing the framework (arXiv:2609.12544).
Constraints#
- No real patient data in the training set.
- 17 languages, including scripts where tokenisers behave very differently.
- Character-exact labels. The model is a BIOES token classifier, so a label that is one character off is a wrong label.
- Small enough to run on-premise at healthcare providers — which ruled out simply calling a large hosted model.
Approach#
Generation is attribute-conditioned: each document is sampled from 12 medical document types, 11 formats and 12 scenarios, with the personal attributes chosen before the text is written. A pool of 9 LLMs writes the documents, so no single model's style dominates.
Every record then passes through 13 deterministic validation gates — regex checks, a character-level offset parser, rule checks and SHA-256 de-duplication. A conservative auto-repair loop fixes what it safely can and re-validates every repaired record; anything else is dropped.
Key technical decisions#
- Deterministic gates over an LLM judge for label correctness. Offsets either match or they don't; a judge model can't guarantee that.
- Several generators instead of one, trading some consistency for diversity.
- Attributes first, text second, so the labels are known by construction rather than recovered afterwards.
Results#
The data trains Meddies-PII-Model, a 350M-parameter (LFM2.5) BIOES token classifier with constrained Viterbi decoding:
| Benchmark | Meddies-PII | Strongest baseline |
|---|---|---|
| 15 external benchmarks (mean exact-match micro-F1) | 0.827 | 0.658 (OpenAI Privacy Filter) |
| 5,100-document adversarial set | 0.878 | 0.540 |
How it was evaluated#
Entity-level exact-match micro-F1: a prediction counts only if both the span and the label match. The external benchmarks were not produced by this pipeline. Details are in the paper.
Limitations — what this does not prove#
- The scores measure a model trained on this data, so they mix data and modelling choices; they are not a pure measure of data quality.
- Public benchmarks may not match the distribution of Meddies' real conversations.
- Synthetic data can carry the generators' own biases.
- A high F1 is not the same as compliance with a specific privacy regulation; de-identification still needs process and human review.
What's next#
meddies-faker extends the pipeline to 20 languages with country-aware personal attributes.