Generative model memorisation is the retention of particular training content in a form that the fitted model can reproduce. In privacy work, the concern is record-specific information that can be exposed through generated outputs or model behaviour. A quant team generating synthetic client transactions or internal research text must therefore assess disclosure separately from whether the samples look realistic.
There is no single universal memorisation threshold. An operational definition must say what counts as a match, what information a user supplies, and how the model is sampled. Carlini et al. (2021) define an extraction-based notion for language models and distinguish content repeated across many training examples from content occurring in very few. That definition is more specific than “the model has learned something about the data”.
Four questions that should stay separate
| Question | What is being assessed? | What does it leave unresolved? |
|---|---|---|
| Are training records duplicated? | Repeated records or substrings in the input dataset. | Whether the fitted model will reproduce them. |
| Does the model reproduce training content? | Exact or explicitly defined approximate matches in its outputs. | Whether a common output depended on a particular person's participation. |
| Can membership be inferred? | Whether a supplied candidate record was used for training. | Whether an unknown record can be recovered. |
| Can training data be extracted? | Recovery of training content through access to the model. | The sensitivity, identity and consequences of each recovered item. |
Membership inference starts with a candidate record and asks whether it was included. It can succeed without generating that record. Shokri et al. (2017) formalise this distinction and explain why membership itself can reveal sensitive information. For a financial example, learning that an already known client record belonged to a confidential distressed-borrower dataset could be consequential even if no transaction values are recovered.
Training-data extraction instead seeks to recover content. A common phrase appearing in both a model output and a training corpus is weak evidence of individual-specific retention: it might also arise from population structure, repeated public text or chance. Conversely, failure to find exact copies does not establish privacy; membership or attribute information can leak without a verbatim output. Stadler et al. (2022) evaluate synthetic-data disclosure through inference tasks rather than treating record distance as a complete privacy measure.
Eight draws from a deliberately copying generator
Consider a fictional generator explicitly designed to output one particular training record with probability on every draw. Assume draws are independent and this probability stays fixed. The probability of at least one exact copy in draws is
With eight draws,
The probability of no copy is about 10.01%. The 89.99% result concerns seeing the specified record at least once; it is not the proportion of outputs that are copies. That per-draw probability remains 25%.
Original analytic illustration of a specified copying mechanism. The curve is a probability calculation, not a measured leakage rate for a trained model.
The mechanism was stipulated, so the calculation cannot diagnose memorisation in an unknown model. It also does not apply unchanged to correlated outputs, changing prompts or sampling without replacement. For a genuinely continuous output distribution, density at a training vector is not a positive probability of exactly reproducing that vector; finite precision and a declared near-match tolerance change the event being counted.
What makes an empirical audit credible?
Begin with the proposed release: one synthetic table, repeated model queries and downloadable weights expose different information. State what the evaluator knows, what counts as sensitive, and the query or sample budget. Assess rare or unusual records as well as averages. These choices follow the threat-model approach in Stadler et al., whose synthetic-table setting differs from unrestricted model access.
Check exact and near duplicates before splitting data. Report duplication levels and compare generated-to-training similarity with generated-to-held-out similarity under the same representation. Kandpal et al. (2022) found that duplication strongly affected regeneration and attack success in the language models they studied. Deduplication is a useful mitigation, not a certificate that all inference channels have disappeared.
Treat an attack score as a statistical measurement. Evaluate on labelled members and nonmembers excluded from attack tuning; report false positives, true positives and uncertainty at a declared threshold. A distribution shift between those groups can make an apparent membership detector recognise the shift instead. Searching many prompts, distance cutoffs or record subsets also creates multiple-testing concerns: preserve a final assessment rather than reporting only the most striking match.
A failed attack establishes the limits of that evaluation's power and access. It does not prove that a stronger attack cannot work. Likewise, low overfitting on an aggregate predictive metric does not settle record-level leakage; Shokri et al. identify overfitting as one contributor, rather than the sole explanation.
For quant use, test utility separately on real held-out observations: for example, whether the intended downstream model retains useful performance across later periods. Time-Ordered Train, Validation and Test Design develops that separation (public preview; full lesson requires membership). Such a result supports the stated use case, not a privacy claim.
Where a formal guarantee enters
Differential privacy constrains how much a randomised mechanism's output distribution can change between neighbouring datasets. For datasets differing by one protected unit, a common definition requires, for every output event ,
The protected unit matters: a transaction-level guarantee does not automatically protect a client's entire transaction history. The parameters, neighbouring-dataset convention and complete data-dependent pipeline must be specified. This is a bound on participation effects, not a promise that no sensitive fact can be inferred from population correlations.
If a trained generator is itself released by a differentially private mechanism, sampling from that fixed generator without returning to private data is post-processing and preserves its guarantee. Retraining on private data requires separate accounting. Dwork and Roth, §2.3 give the definition, post-processing result and basic composition principles. Calling data “synthetic”, removing names, or finding no copied rows supplies none of those formal conditions.
Sources
- Carlini et al. (2021), Extracting Training Data from Large Language Models, §3.1
- Shokri et al. (2017), Membership Inference Attacks Against Machine Learning Models, §§III–V
- Kandpal, Wallace and Raffel (2022), Deduplicating Training Data Mitigates Privacy Risks in Language Models, §§3–4
- Stadler, Oprisanu and Troncoso (2022), Synthetic Data – Anonymisation Groundhog Day, §§3–4
- Dwork and Roth (2014), The Algorithmic Foundations of Differential Privacy, §2.3