Data Poisoning in AI: How Training Data and Knowledge Sources Can Be Manipulated

AI systems learn or retrieve information from data. That makes the integrity of the data pipeline a security boundary, not just a data-quality concern.

A poisoned record does not need to look like malware. It may be a mislabeled training example, a deliberately crafted document, a manipulated feedback record, or content placed in a source that an AI pipeline later trusts. If that material influences training or another persistent knowledge process, the resulting system can learn or retrieve behavior that its operator never intended.

NIST’s Adversarial Machine Learning taxonomy treats poisoning as a major adversarial machine learning attack family and discusses it for predictive and generative AI. OWASP’s LLM04:2025 Data and Model Poisoning similarly identifies manipulation of pre-training, fine-tuning and embedding data as an integrity risk.

The useful security question is therefore not simply, “Is our dataset clean?” It is: who can influence the information that changes model behavior, how is that information validated, and can we trace a suspicious behavior back to its source?

What is data poisoning in AI?

Data poisoning is deliberate manipulation of data used by an AI or machine-learning process so that the system behaves differently from what its owner intended. NIST classifies poisoning by factors such as the attacker’s goal, knowledge and capability. The exact mechanism depends on the learning system.

In supervised learning, an attacker might influence examples or labels. In a generative AI pipeline, the affected material can include pre-training corpora, instruction-tuning data, preference or feedback data, and other datasets used to adapt a model. OWASP also includes embedding data in its LLM04 discussion because applications increasingly depend on externally sourced information that is transformed and stored for later use.

Poisoning is primarily an integrity problem. The attacker is trying to corrupt what the system learns, trusts or later uses as knowledge. The visible result may be broad degradation, a targeted wrong answer, an unwanted bias, or behavior that appears only when a particular condition is present.

That last case is important. A model can appear normal during ordinary use while behaving differently for a trigger or narrow class of inputs. NIST’s work on explaining poisoned models discusses poisoned models with malicious triggers, illustrating why average accuracy alone is not sufficient evidence that a model is clean.

Where can poisoning enter the AI lifecycle?

There is no single “training dataset” in many modern AI systems. Information passes through several stages, often involving third parties and automated ingestion.

Pre-training data

Foundation models can be trained on very large and diverse corpora. NIST notes that the scale and diversity of sources creates a broad attack surface. Its 2025 report gives an instructive supply-chain example: when a dataset is represented by URLs, an attacker who later controls a referenced domain may be able to replace the content served from it.

This does not mean that arbitrary web content automatically controls a model. The effect depends on collection, filtering, deduplication, weighting, training and many other factors. The security lesson is narrower: provenance and source control matter even before model training starts.

Fine-tuning and instruction data

Fine-tuning uses a smaller, more task-specific dataset to alter model behavior. That makes access to the fine-tuning pipeline particularly sensitive. A malicious insider, compromised data preparation process, or untrusted external dataset can introduce examples that shift behavior.

The smaller scale can make governance easier than internet-scale pre-training, but only if the organisation actually records where examples came from, who approved them and which version entered a training run.

Feedback and human-generated data

Human feedback is still data. If a system uses ratings, preference pairs, corrections or moderation decisions to improve later models, the integrity of that channel matters. NIST explicitly notes that poisoning can affect stages including instruction tuning and reinforcement learning from human feedback.

A useful design therefore separates ordinary product feedback from trusted training approval. “A user submitted it” should not automatically mean “the next model should learn from it.”

Embeddings and knowledge stores

Retrieval-augmented generation (RAG) introduces another persistent data path. Documents are ingested, transformed into embeddings, indexed and later retrieved into a model’s context. If an attacker can add or alter trusted documents, the application’s answers can be manipulated even though the underlying foundation model was never retrained.

This is often described as knowledge-base or retrieval poisoning. It is related to data poisoning because the trusted information store is corrupted, but the technical effect is different from modifying model weights. Our guides to RAG security and secure RAG architecture examine that application layer in more detail.

Training poisoning, retrieval poisoning and prompt injection are not the same

These terms are easy to mix because all three can involve malicious text.

The distinction matters when choosing controls. A prompt filter cannot restore the integrity of a compromised training set. Dataset checks cannot by themselves stop an agent from treating a malicious webpage as an instruction. See Indirect Prompt Injection for that separate attack path.

What can an attacker try to achieve?

Poisoning is not limited to making a model generally “worse.” Different objectives require different tests.

An attacker may try to reduce overall performance, influence a particular class of predictions, insert unwanted associations, create trigger-dependent behavior, or manipulate downstream decisions that rely on model output. OWASP notes consequences including degraded performance, biased or toxic content, backdoors and downstream exploitation.

A targeted attack can be harder to notice than broad degradation. If a poisoned model fails only on a rare trigger, an aggregate benchmark may barely move. This is why security evaluation should include threat-driven cases rather than relying only on one overall accuracy or quality score.

A practical threat model for an AI data pipeline

Consider an organisation that fine-tunes a support model using approved historical tickets and also maintains a RAG knowledge base from internal documentation.

The first step is to map influence, not just storage. Ask who can:

1. add a ticket that becomes a training candidate;
2. approve or relabel examples;
3. modify preprocessing code;
4. change the dataset manifest;
5. publish documents into the RAG source;
6. rebuild embeddings or indexes;
7. approve a model or index for production.

Now separate those permissions. A support user who can create a ticket should not automatically be able to approve that ticket for training. A document contributor may legitimately edit a draft, but production knowledge ingestion can still require source identity, scope and review.

The diagram above models this as a series of trust boundaries. Controls are strongest when they prevent one compromised stage from silently controlling every later stage.

Defending the training and fine-tuning pipeline

There is no universal detector that can prove a large dataset is free of poisoning. NIST explicitly discusses limitations and open challenges in adversarial ML mitigation, so defence should be layered.

Protect provenance. Record where data came from, when it was collected, what transformation occurred and which dataset version was used. Immutable or append-only audit records make later investigation much more practical.

Control write access. Treat training datasets, labeling systems, object stores, preprocessing jobs and model registries as production security assets. Use least privilege and separate roles for contribution, approval and release where the risk justifies it.

Validate before learning. Schema validation catches malformed data, but security review must go further. Look for unexpected label distributions, duplicate or near-duplicate examples, unusual source concentration, outliers, abrupt changes in data composition and samples that conflict with trusted references. These signals are investigative leads, not proof of an attack.

Version datasets and code together. A reproducible training run should identify both its data snapshot and transformation code. Otherwise, discovering a suspicious model does not tell you what actually produced it.

Evaluate targeted behavior. Maintain clean holdout data and security-focused test cases that are independent from the training pipeline. When the threat model includes trigger-based behavior, test plausible triggers and sensitive classes rather than checking only average performance.

Protect release decisions. A model artifact should not move to production merely because training completed. Record evaluation results, model identity, dataset lineage and approval before promotion.

Defending a RAG or knowledge pipeline

A RAG system needs many of the same integrity principles, but it also needs runtime controls because retrieved content reaches the model during inference.

Start by deciding which sources are authoritative for each use case. An internal policy assistant should not silently treat arbitrary user uploads as equivalent to approved policy documents. Preserve source metadata through chunking and indexing so a retrieved statement can be traced back to a document and owner.

Apply authorization before retrieval, not after the model has already seen the content. A user’s query should only search material that user is permitted to access. For high-impact answers, expose citations or provenance so a person or downstream control can inspect the supporting source.

Finally, remember that a poisoned document can contain both false information and malicious instructions. Retrieval integrity therefore works alongside prompt-injection controls. The secure RAG architecture guide explains why retrieved text should remain untrusted even when it comes from an indexed source.

A safe validation pattern

A defensive team can test pipeline integrity without creating a harmful payload. In a non-production environment, add a clearly marked synthetic document such as:

TEST RECORD - NOT AUTHORITATIVE
control_id: DP-INTEGRITY-001
statement: The approved test value is ORANGE-17.
source_class: untrusted-test

Then verify the controls around it. Does ingestion preserve the untrusted-test classification? Can an ordinary contributor promote it to an authoritative collection? Does retrieval show its provenance? Is it excluded from a fine-tuning export unless an authorised reviewer approves it?

The point is not whether the model repeats ORANGE-17. The useful evidence is whether the pipeline preserves trust metadata and enforces the intended approval boundary from source to deployment.

What should you monitor after deployment?

Pre-deployment validation cannot anticipate every poisoned behavior. Production monitoring should therefore look for changes that matter to the system’s actual purpose.

Useful signals can include sudden shifts in error categories, abnormal behavior for specific input classes, unexplained changes after a model or knowledge-index update, repeated retrieval from unexpected sources, and security test regressions. Keep enough version information to connect an incident to the model, prompt configuration, dataset and index that were active at the time.

Monitoring also needs a response path. If a suspicious source is found, the organisation should be able to quarantine it, identify which artifacts consumed it, rebuild affected indexes or models where necessary, and retest before restoring service.

Common mistakes

One mistake is treating data poisoning as a problem that a model provider alone must solve. Application teams create their own fine-tuning sets, feedback loops and knowledge stores, so they also create integrity boundaries.

Another is assuming that “trusted internal data” cannot be poisoned. Internal sources can be changed by compromised accounts, excessive permissions, automation errors or malicious insiders. Trust should come from controlled provenance and authorization, not simply a network location.

A third mistake is confusing sanitization with certainty. Removing suspicious strings may be useful in a narrow pipeline, but poisoning is a semantic problem as well as a syntactic one. A perfectly valid sentence can still be deliberately false or strategically mislabeled.

The deeper security lesson

AI data poisoning is a supply-chain and integrity problem as much as a model problem. The model is the visible component, but the security decision begins earlier: which information is allowed to influence it, under whose authority, with what evidence, and through which transformations?

A strong AI pipeline makes those answers inspectable. It limits who can change influential data, preserves provenance, separates contribution from approval, evaluates targeted behaviors and keeps enough lineage to recover when something goes wrong.

That approach will not make poisoning impossible. It does make silent manipulation harder and investigation far more realistic.

Sources and further reading

• NIST AI 100-2e2025: Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
• NIST: Explaining Poisoned AI Models
• OWASP LLM04:2025 Data and Model Poisoning
• OWASP Machine Learning Security Top 10: Data Poisoning Attack