Data Poisoning

Gu, Dolan-Gavitt, and Garg introduced the first systematic treatment of backdoor attacks on neural networks in 2017, naming the technique BadNets. The core insight is that a neural network trained on a dataset containing two patterns simultaneously will learn both: the real classification task and the attacker-specified trigger-to-target association. The network does not know these are different kinds of patterns. It treats both as legitimate signals from the training data.

The attack requires access to the training data, which is feasible if training data is collected from public sources, if the attacker is an insider, or if the attacker controls a data provider in the training pipeline.

The critical property is that the model's accuracy on clean test sets is completely unchanged. The backdoor only activates when the trigger is present. Standard evaluation, which measures accuracy on a clean test set, sees a perfectly normal model. This is why backdoor attacks are fundamentally a training data integrity problem, not a model evaluation problem. You cannot find a backdoor by testing the model if you do not test it with the trigger.

The choice of trigger type determines how detectable the attack is, what kind of detection method can find it, and how it gets delivered in deployment. For AI agents, the natural language trigger type is the most operationally relevant.

Standard data poisoning changes both the input and the label. If a security team manually reviews training data for mislabelled examples, standard poisoning can be caught: an image of a stop sign labelled "speed limit" is obviously wrong.

Turner et al. addressed this in 2018 by showing that a backdoor can be embedded without ever mislabelling a training example. Every label in a clean-label attack is correct. The attack works through the image features instead.

The mechanism works because the adversarial perturbation moves the image into a region of feature space where the model already associates the input with class B. When the trigger is also present, the model sees an input that looks like class B in feature space plus the trigger signal, and strongly predicts class B. The human reviewer sees an image that looks like class A with the correct label.

Clean-label attacks are harder to defend against because standard countermeasures that look for mislabelled examples do not help. Defending against clean-label attacks requires inspecting the statistical properties of the image features themselves, not just the labels.

Large language models are trained in stages. Pre-training on web-scale text gives the model broad capabilities. Instruction tuning on curated instruction-response datasets makes it follow instructions reliably. RLHF further aligns it with human preferences. Each stage is an attack surface, and the attack surface for LLMs is significantly larger than for image classifiers because the training data is sourced from public web content at unprecedented scale.

The instruction tuning attack is the most immediately relevant for practitioners building AI agents. Most teams fine-tune a foundation model on their own instruction data. If any portion of that instruction data is sourced from public datasets, user submissions, or scraped content, the possibility of poisoned examples exists. 100 poisoned examples in 10,000 is a 1% rate, which is well within the range demonstrated as effective for backdoor attacks.

The most unsettling property of a well-constructed backdoor is that the model genuinely is performing correctly, for every input it was evaluated on. Standard evaluation does not find backdoors not because the evaluation method is poorly designed, but because it is evaluating the right thing on the wrong inputs. The evaluation question is "does this model classify correctly?" and the answer is yes, for every example that does not contain the trigger.

This creates a fundamental detection challenge. If you do not know what the trigger looks like, you cannot construct a test set that includes it. And if you could construct such a test set, you would already have detected the backdoor through some other means. Solving the evaluation gap requires methods that do not depend on knowing the trigger in advance, which is exactly what activation clustering, spectral signatures, and Neural Cleanse provide.

Four detection methods have been established in the literature, each working at a different stage of the ML pipeline and with different capability-limitation tradeoffs. None is universally effective. A production system should combine multiple methods.

Defences against data poisoning operate at three levels: before training (data sanitisation and provenance), during training (differential privacy, which is covered in detail in section 08), and after training (detection methods from section 06 combined with targeted fine-tuning to remove detected backdoors).

For LLMs specifically, instruction tuning dataset provenance is the most important control. Track every source that contributed instruction examples, maintain a per-source hash, and run trigger-phrase scanning against any public or user-submitted content before including it in the fine-tuning set.

Certified defences provide a formal guarantee: if the fraction of poisoned training data is below a threshold T, the model's prediction on any input is guaranteed to match the prediction of a model trained on clean data. Work by Rosenfeld et al. (2020) and others provides such guarantees for certain model architectures and training procedures. The tradeoff is that the training procedure is constrained, which may reduce model performance relative to standard training.

Data poisoning works by injecting examples with high influence on the model's learned behaviour. The poisoned examples push the decision boundary in a specific direction over many training steps. Differential privacy constrains how much any individual example can push the boundary.

Abadi et al. at Google introduced DP-SGD (Differentially Private Stochastic Gradient Descent) in 2016. DP-SGD adds two modifications to standard training: gradient clipping (each example's gradient is clipped to have at most norm C before accumulation) and noise addition (Gaussian noise is added to the accumulated gradient before the update step). Together these provide an epsilon-differential privacy guarantee: no single training example can change any model output by more than a bounded amount.

DP-SGD is not a complete solution to data poisoning on its own. At the epsilon values needed for meaningful poisoning resistance, model accuracy typically drops. But it provides a formal mathematical bound on the influence of any individual training example, which is a stronger guarantee than heuristic detection methods alone. For high-stakes deployments (medical, financial, legal), combining DP-SGD with data provenance tracking and post-training detection provides a layered defence with both formal and empirical coverage.

For LLM fine-tuning, DP-SGD introduces additional complexity because the noise calibration depends on the number of training examples and the privacy budget across all training steps. Libraries like Google's tensorflow-privacy and Opacus (PyTorch) provide DP-SGD implementations that handle these details.

Before training a model on data that will be used in a production AI agent, verify the following controls are in place.