Can AI Models Leak Training Data? Understanding Memorisation and Data Exposure

Large language models are designed to learn patterns from data, not to behave like searchable copies of their training sets. Even so, research has shown that some models can memorise specific training examples and later reproduce them under the right conditions. That creates a real privacy and security question: can someone recover sensitive information from a trained model?

The answer is yes, under some circumstances. But the details matter. A model producing a familiar sentence does not automatically prove that the sentence came from its training data. A user revealing information in the current conversation is not the same as training data leakage. A retrieval system returning a confidential document is also a different problem.

This article focuses on memorisation and training data extraction, where information that influenced model training becomes recoverable from the model itself.

NIST’s current Adversarial Machine Learning taxonomy classifies privacy attacks as a major attack family and defines training data extraction as recovering training data from a generative model by prompting it with specific inputs. Earlier research published at USENIX Security 2021 demonstrated that verbatim training examples could be recovered from GPT-2 through carefully selected queries.

Memorisation is not the same as learning a general pattern

A language model is useful because it learns statistical relationships that generalise beyond individual examples. If it has seen many examples of valid email addresses, it can learn the structure of an email address without needing to store a specific person’s address as an explicit database record.

Memorisation is different. It occurs when the model retains enough information about a specific training example that the example, or an unusually close version of it, can be reproduced later.

This distinction is important because generalisation is normally the goal of training, while unnecessary memorisation can create privacy exposure.

Rare or unusual sequences deserve particular attention. A common phrase appearing thousands of times is not very informative if a model repeats it. A unique customer record, API credential, private message or unusual identifier is different because reproducing it may reveal information that should never have been available to an external user.

Google researchers studying unintended memorisation developed methods for inserting artificial rare sequences, often called canaries, into training data and measuring how strongly a model memorised them. Their Secret Sharer research showed that such testing can help quantify memorisation risk before deployment.

What is a training data extraction attack?

A training data extraction attack tries to recover specific examples from the data used to train a model.

The attacker typically does not receive direct access to the original training dataset. Instead, they interact with the trained model and search for prompts or generation strategies that cause memorised sequences to appear.

The 2021 USENIX work by Nicholas Carlini and co-authors demonstrated this against GPT-2. The researchers generated many candidate outputs and then used signals that helped identify unusually memorised text. They were able to recover hundreds of verbatim sequences from the model’s training data, including publicly available personal information, code, conversations and unusual identifiers.

That study is important because it showed that a generative model can act as an unexpected information channel. The data is not exposed through a normal database query. It is exposed because the training process encoded enough information about particular examples into the model parameters.

NIST now includes training data extraction in its formal adversarial machine learning terminology, which reflects how the issue has moved from a research curiosity into a recognised privacy threat category.

What makes some records more likely to be memorised?

There is no simple rule that says a particular record will definitely be memorised. Model architecture, dataset size, repetition, training procedure, model capacity and other factors all influence the result.

However, research has repeatedly shown that rare and repeated sequences can be especially important.

Repetition matters because the training process sees the same information more often. Google research on deduplicating language model training data found that duplicated examples were associated with more memorised generations. Deduplication therefore has both model-quality and privacy value in some training pipelines.

Rarity also matters because an unusual sequence can stand out strongly. A random identifier, private message fragment or secret token may be much more revealing than a common sentence if it is reproduced.

The practical lesson is not that every repeated record will leak. It is that organisations should not assume that large datasets automatically hide sensitive examples through scale.

Memorisation, membership inference and model extraction are different attacks

Several AI privacy attacks are often grouped together even though they answer different questions.

NIST’s 2025 taxonomy separates these concepts because the attacker objective and mitigation strategy differ.

For example, a prompt extraction issue may reveal a hidden system instruction that was added at runtime. That does not mean the model memorised the system prompt during training. Similarly, a membership inference result may suggest that a record influenced training without recovering the full record itself.

Using precise terminology makes incident response much easier because the team can investigate the correct data path.

Training data leakage is also different from RAG leakage

Modern AI applications often combine a foundation model with Retrieval-Augmented Generation, or RAG. In a RAG system, external documents are retrieved at runtime and inserted into the model context.

If a confidential document is incorrectly retrieved and shown to an unauthorised user, the root problem is normally access control or retrieval design, not model memorisation. The document may never have been part of model training at all.

This distinction is covered in more depth in RAG Security Explained. It also matters for incident containment. A RAG exposure may be fixed by correcting document permissions, retrieval filters or tenant isolation. A true training-data exposure may require retraining, model replacement or stronger privacy controls around the training pipeline.

The same reasoning helps separate this topic from data poisoning. Poisoning targets model integrity by changing what the model learns. Training data extraction targets confidentiality by recovering information that the model learned too specifically.

Can private fine-tuning data leak too?

Yes, that is a meaningful risk to evaluate.

Fine-tuning often uses much smaller datasets than pre-training. Those datasets may include organisation-specific conversations, support cases, internal documents or other records that are more sensitive than public web text.

Smaller datasets can also make individual examples more significant to the training process. If the same sensitive record is repeated, or if fine-tuning strongly adapts the model to a narrow corpus, unintended memorisation may become more concerning.

The correct response is not to assume that every fine-tuned model leaks its dataset. Instead, teams should treat private fine-tuning data as a security boundary and test the resulting model before release.

Google DeepMind research published at ICLR 2024 demonstrated a related risk called neural phishing, where an attacker who can influence training data can create conditions that later help extract targeted sensitive information. This is a reminder that privacy and integrity threats can interact. An attacker may first manipulate training and then exploit memorisation during inference.

What should never be placed into training data without a clear need?

The safest secret is usually the one the model never receives.

Before training or fine-tuning, organisations should identify data that has no legitimate reason to influence the model. Examples include:

• passwords, private keys, API tokens and session secrets;
• full payment card data or authentication codes;
• unnecessary personal identifiers;
• private customer or employee messages unrelated to the training objective;
• confidential documents that are not required for the intended task;
• production credentials accidentally captured in logs, tickets or source code.

Removing such material is stronger than hoping an output filter will catch every future disclosure.

This principle also connects to protecting sensitive data and credentials when using AI tools. Data minimisation should happen before information reaches model training, fine-tuning or external AI services whenever possible.

A practical control strategy

Reducing training data exposure requires controls across the full model lifecycle. No single mitigation solves every case.

1. Minimise and classify data before training

Start by asking whether each data source is actually needed. Sensitive fields should be removed, masked or transformed when they do not contribute to the learning objective.

Maintain provenance for important datasets so that teams know where records came from, what processing was performed and which model versions consumed them. This becomes critical if a later incident requires retraining or deletion.

2. Detect secrets and high-risk personal data

Automated scanning can identify many obvious risks before training, including common credential formats, private keys and structured personal identifiers. Human review may still be necessary for context-sensitive information.

Secret scanning should not be treated as perfect. A proprietary access token may not match a known pattern, and ordinary text can still contain sensitive business information.

3. Deduplicate where appropriate

Removing unnecessary duplicate records can reduce repeated exposure to the same sequence and may also improve training quality. Deduplication needs to be designed carefully because two similar records are not always redundant.

Keep enough provenance to explain what was removed and why.

4. Consider privacy-aware training methods

Differential privacy can provide formal privacy guarantees when it is correctly designed and implemented. In machine learning, techniques such as differentially private training can limit how much influence any individual training example has on the resulting model.

This can reduce memorisation risk, but it introduces engineering trade-offs. Privacy parameters, training stability, model quality and computational cost all matter. Differential privacy should therefore be treated as a measurable design choice, not a checkbox.

5. Test for memorisation before release

High-risk models should be evaluated specifically for data exposure, not only for accuracy and helpfulness.

One safe approach is to add synthetic canary strings to controlled training data and test whether those rare sequences become extractable. Because the canaries are artificial, the evaluation does not intentionally expose real customer secrets.

Teams can also probe whether known sensitive-pattern classes are reproduced, but testing should be designed so that evaluators do not create a new privacy incident while searching for one.

6. Add runtime safeguards

Runtime controls can reduce exploitation even when training-time controls are imperfect. Useful measures can include rate limits, anomaly detection, output screening for known secret formats, logging and restrictions on bulk automated querying.

These controls are secondary defenses. If the underlying model has memorised sensitive information, filtering only the output does not remove that information from the model.

7. Plan for model withdrawal and retraining

If a model is confirmed to expose sensitive training records, incident response may require more than changing a prompt.

The team should identify the affected model versions, investigate the source dataset, preserve relevant evidence, assess who could query the model and determine whether a safer checkpoint exists. Depending on the severity, the correct action may be to withdraw the model and retrain from a sanitised dataset.

Machine unlearning is an active research area, but organisations should not assume that deleting a row from the source dataset automatically removes its influence from an already trained model.

A useful way to threat-model the problem

When reviewing an AI system, ask four questions:

1. What sensitive information can enter training or fine-tuning?
2. Which records are rare, repeated or unusually identifying?
3. Who can query the resulting model, and at what scale?
4. What evidence would prove that a model version is safe enough to release?

Those questions connect data governance to security testing. They also help avoid a common failure mode where the privacy team reviews the dataset but nobody tests the behaviour of the trained model.

For a broader view of model, dataset and dependency trust, see AI Supply Chain Security. The supply chain perspective is useful because training data exposure is easier to manage when datasets and model versions are traceable from ingestion through deployment.

Do not over-interpret a suspicious model response

A model returning a real-looking phone number, email address or code snippet is not proof of training data leakage by itself.

Generative models can produce plausible information that was never present in training. They can also reproduce public facts that occur in many places. A rigorous investigation therefore needs evidence connecting an output to a known training record or a controlled extraction test.

This is especially important when discussing security incidents publicly. It is easy to mistake coincidence, public information or retrieval output for memorisation. Claims about leaked training data should be supported by reproducible evidence, dataset knowledge and careful analysis.

The practical security lesson

AI models should not be treated as perfect one-way transformations that permanently hide their training inputs.

Research and current NIST guidance show that privacy attacks against trained models are real, and that generative models can sometimes reveal specific training examples. The strongest response is lifecycle-based: minimise sensitive input, maintain data provenance, reduce unnecessary duplication, use privacy-aware training where justified, test for memorisation, control high-volume querying and be prepared to replace a model that fails privacy evaluation.

The goal is not to guarantee that a model remembers nothing. The goal is to prevent sensitive information from becoming recoverable in ways that create real harm.

Sources and further reading

• NIST AI 100-2e2025: Adversarial Machine Learning, A Taxonomy and Terminology of Attacks and Mitigations
• NIST Glossary: Training Data Extraction
• USENIX Security 2021: Extracting Training Data from Large Language Models
• Google Research: The Secret Sharer, Evaluating and Testing Unintended Memorization in Neural Networks
• Google Research: Deduplicating Training Data Makes Language Models Better
• Google DeepMind: Teach LLMs to Phish, Stealing Private Information from Language Models