AI Threat Modeling

Traditional threat modeling, whether you use STRIDE, PASTA, or LINDDUN, was designed for software systems with deterministic inputs and outputs. You map data flows, identify trust boundaries, enumerate threats at each boundary, and rank them. The methodology works. But AI systems have three characteristics that traditional threat modeling does not address well.

The model itself is an attack artifact. In traditional software, the code is the artifact and it can be patched when a vulnerability is found. In AI systems, the model weights are also an artifact. Weights can be poisoned during training, backdoored invisibly, or extracted through systematic API queries. There is no "patch" equivalent for a compromised model: you retrain, which costs time and money.

Natural language inputs cannot be fully sanitised. SQL injection is fixed by parameterising queries. The fix is complete and the threat class is closed. Prompt injection has no equivalent fix because the model must read natural language to function and cannot reliably separate instructions from data in that natural language. Every improvement is a partial control, not a closure.

The training pipeline is a new attack surface. Traditional software has a build pipeline. If an attacker can compromise the build, that is serious but well-understood. AI systems have a training pipeline where the data that flows through it shapes the model's behaviour forever. Poisoning training data changes the model's behaviour in ways that can be invisible at evaluation time and only trigger on specific inputs.

These three differences do not break the STRIDE methodology. They require adding AI-specific threat examples to each STRIDE category and adding new components and trust boundaries that traditional data flow diagrams do not include.

AI threat modeling follows the same five steps as traditional threat modeling, but each step has AI-specific content. The process is designed to produce one output: a prioritised list of threats with associated controls, so you know what to build first.

A data flow diagram for an AI system has more components than a traditional web application. The diagram below shows the components for a typical LLM application with RAG and tool access. Every component is a potential attack surface. Every arrow is a data flow that crosses a trust boundary or stays within one.

The components most commonly left off AI data flow diagrams are the prompt construction layer (where the system prompt and retrieved context are assembled before the model sees them) and the model-to-tool gateway (where model output is translated into actual API calls). Both are critical for threat modeling because both are points where an injection attack can pivot from causing the model to say wrong things to causing the model to do wrong things.

A trust boundary is a line where data moves from one zone of trust to another. Threats arise at trust boundaries because that is where data passes between different owners, different trust levels, and different validation rules. If you miss a trust boundary, you miss the threats that cross it.

AI systems have trust boundaries that traditional web applications do not have. The four most important ones:

User to application. User input is always untrusted. This is the same as in traditional web applications. The difference is what "malicious input" looks like: in traditional apps, you scan for SQL fragments, HTML tags, and shell metacharacters. In AI apps, malicious input is natural language that redirects the model's behaviour. There is no equivalent of a regex that catches all prompt injection.

Retrieval to context window. Documents retrieved from a vector database and placed into the model's context are only as trusted as the source that added them to the index. If the index includes content from user uploads, web crawls, or third-party sources, that content is partially untrusted. An attacker who can add a document to the index can inject instructions into the model's context without the user sending anything malicious.

Model output to tool execution. The model's text output is plausible but not guaranteed to be safe to execute. When model output is parsed as a tool call and executed against real APIs, that execution crosses a critical trust boundary. The model's reasoning may have been compromised by an injection in the context. Validating tool calls against an explicit policy before execution is the primary defence here.

Agent to external APIs. All external API responses must be treated as untrusted. An API that an agent calls might be compromised or might return content that contains further injection instructions. An agent that processes API responses without validation is exposed to indirect injection from the API layer.

STRIDE is a threat taxonomy, not a methodology. It gives you six categories to ask about for each data flow. Applied to AI systems, each category has specific AI threat examples that traditional STRIDE lists do not include.

After generating your raw threat list from STRIDE, map each threat to a MITRE ATLAS tactic. This does two things. It tells you whether the threat is part of a documented adversary pattern with known detection logic. And it gives you a common vocabulary to use with your security operations team, so your threat model connects directly to what they monitor in the SIEM.

The eight ATLAS tactics most commonly relevant to deployed LLM applications are listed below with the system component they most often target.

The system: a customer service chatbot for an e-commerce company. It has access to a knowledge base via RAG (product documentation, return policies). It can call two tools: a CRM read API to look up the customer's order history, and a refunds API to initiate refunds up to a configured limit. Users authenticate with a session token but the model itself does not have per-user authorisation logic.

This is a real deployment pattern. Many production LLM applications have similar components. The threat model below applies steps 1 to 4 to this specific system before producing the threat register in the next section.

Trust boundary analysis for this system:

The user-to-application boundary is the main injection surface. Malicious user input can attempt direct prompt injection. The retrieval-to-context boundary is a RAG poisoning surface: if the knowledge base is ever updated with user-submitted content (such as product reviews or support tickets), those are untrusted documents in the index. The model-to-tool boundary is the highest-risk boundary: the refunds API can cause real financial harm if an injection convinces the model to call it without a legitimate user request. The CRM API is a data exfiltration surface: an injection could cause the model to return other customers' order data.

System components relevant to threat modeling: user session, API gateway with authentication, prompt construction layer assembling system prompt plus retrieved docs plus user query, vector database containing product and policy documentation, LLM inference, tool gateway (the AgentIQ layer), CRM read API, refunds API.

The threat register is the output of the threat modeling process. It takes every threat identified in steps 2 to 4 and scores each one on likelihood and impact. Multiply the two scores to get a priority number. Address the highest-priority threats first. The register below shows the threats identified for the customer service LLM worked example.

Scoring is 1 to 5 for both dimensions. Likelihood: 1 = requires advanced attacker, 5 = any user can trigger with minimal effort. Impact: 1 = inconvenience, 5 = financial harm, regulatory breach, or reputational damage.

The threat register tells you what to fix and in what order. The top three threats in the worked example are all scored Critical. The controls for each:

Prompt injection triggers refunds API (score 25). The primary control is enforcing deny-by-default tool access at the tool gateway. The refunds API should only be callable when the authenticated user has initiated a refund request through an explicit UI action, not through natural language alone. AgentIQ's policy engine can enforce this: a policy that requires a verified user intent signal before the refunds tool call is permitted.

RAG index poisoning (score 20). Two controls. First, restrict who can add content to the knowledge base index. If user-submitted content is never in the index, the poisoning surface does not exist. Second, validate retrieved documents before adding them to the context by running them through a content classification step. DiscoveR's RAG poisoning category tests whether your current index is vulnerable.

CRM data exfiltration (score 20). AgentIQ's output classification checks whether model outputs contain PII patterns from the CRM before the response reaches the user. AgentID's capability-scoped tokens ensure the CRM read tool is bound to the authenticated user's scope, so the model cannot request a different customer's data regardless of what the injection says.

The pattern is consistent: for each high-priority threat, the control is either a policy at the tool gateway (AgentIQ), a restriction on what data enters the context (data governance + RAG hygiene), or an output check that catches leakage before it leaves the system (AgentIQ output classification).

A threat model is a hypothesis. It says: we believe these threats are likely and these controls will mitigate them. DiscoveR is how you test whether the hypothesis is correct.

After completing your threat model, the prioritised threat register becomes the input to a DiscoveR scan. The top threats in the register map directly to DiscoveR attack categories: prompt injection maps to the injection and jailbreak categories, RAG poisoning maps to the RAG poisoning category, tool abuse maps to the tool abuse category, CRM exfiltration maps to the data exfiltration category.

The DiscoveR scan result tells you three things. First, which theoretical threats from the model are actually exploitable in your current deployment. Second, whether the controls you implemented since the last scan have reduced your exposure. Third, whether there are exploitable threats your threat model did not identify, which tells you where your model needs updating.

This is the feedback loop: threat model identifies threats, controls are implemented, DiscoveR validates whether the controls work, findings update the threat model. Run this cycle before each major deployment, after significant changes to the system, and on a regular schedule (quarterly is the minimum most teams should aim for in production).

Module 05 closes Track 1 with AI governance and compliance. It maps NIST AI RMF, ISO 42001, and the EU AI Act to practical obligations and explains how a threat model like the one in this module produces evidence those frameworks require. After Track 1, choose the path that matches what you are building.