Zulia Shavaeva← Back to Zulia
Free AI and security guide/Learn

Learn / AI system threat modelling and secure architecture

AI system threat modelling and secure architecture

Mapping how an AI or agentic system can be attacked and designing the architecture and controls to reduce that risk.

Researched on 2026-09-26 with AI assistance. Links and summaries can change; verify details with the original source. Not yet reviewed by a person.

What it is

AI system threat modelling is the practice of drawing how an AI or agentic system handles data, models, tools and identities, then enumerating how each part can be attacked and which controls reduce that risk. Microsoft's AETHER guidance frames it as an extension of normal SDL threat modelling, with new questions to ask during security design reviews of products that use or embed ML.8

The scope is wider than the application. NCSC's guidelines state that AI systems carry novel vulnerabilities, described as adversarial machine learning, alongside standard cyber security threats, and that security has to be treated as a lifecycle requirement rather than a development phase task.3

Google's SAIF map splits AI development into data, infrastructure, model and application, and maps each risk to the controls that address it, with governance and assurance controls applying across every risk and stage. That four area split is a workable skeleton for a first architecture review.910

Secure architecture then means choosing structural defences, not just filters. The design patterns paper argues for constraining what an agent is allowed to do so it cannot be steered into arbitrary tasks, and CaMeL shows a system layer that separates control flow from untrusted data and enforces policy at tool call time.56

Why postings ask for it

22 of 48 postings (46%) ask for this skill, and it is the dominant ask in the design-side clusters: 86% of the 7 AI Security Architect postings and 62% of the 16 AI/Agent Security Engineer postings.P

43% of the 7 AI Governance / GRC postings ask for it, because risk functions need a technical map before they can assign controls; NIST's AI RMF Core is organised into four functions for exactly that structuring work.P1718

Only 22% of the 9 red team postings and none of the 5 consulting postings ask for it, so the differentiator for testers moving into architect or engineer roles is turning findings into design changes, which is what the agent design pattern literature covers.P5

Concepts you should be able to explain

If you can say each of these out loud in two minutes, with an example, you are ready for the technical part of an interview on this skill.

Trust boundaries when data becomes instructions

LLM-integrated applications blur the line between data and instructions, so any retrieved document, web page or tool output crosses a trust boundary the moment the model reads it. The indirect prompt injection paper shows retrieved content acting as arbitrary code execution over the application's own functionality and API calls. Your diagram must therefore label every content source as trusted or untrusted before you list threats.7

Prompt injection, direct and indirect

OWASP lists prompt injection as LLM01, where user prompts alter the model's behaviour. The indirect variant lets an attacker with no interface to the app inject instructions into data the app will later retrieve, which was demonstrated against real systems including Bing's GPT-4 powered chat and code completion engines. The design patterns paper treats both as one class, since the defence is structural either way.175

Excessive agency and tool permissions

OWASP LLM06 covers systems granted more agency, permissions or autonomy than the task needs. In architecture terms every tool, credential and write path the agent can reach is part of the blast radius of a single successful injection. PortSwigger's excessive agency lab is the cheapest way to demonstrate the failure to a sceptical engineering audience.113

Output handling and exfiltration channels

OWASP LLM05 is improper output handling: insufficient validation or sanitisation of model output before it reaches another component. A concrete case is image markdown injection in Bing Chat, where text on a web page such as a user comment or advert caused conversation data to be exfiltrated through a rendered image URL. Rendering, link following and downstream API calls are all sinks that belong in the model.112

Adversarial ML taxonomy beyond prompts

NIST AI 100-2e2025 sets out a hierarchy covering ML method types, lifecycle stages of attack, and attacker goals, objectives, capabilities and knowledge, with data poisoning, evasion and privacy breach among its keywords. Using its terms stops a threat model collapsing into a prompt injection list. Microsoft's guidance makes the same point from the engineering side by placing training data stores inside threat modelling scope.48

Supply chain and multi actor responsibility

OWASP LLM03 covers supply chain weaknesses in models, data and dependencies. NCSC notes that modern AI supply chains contain many actors, so a provider building on external APIs inherits risk it does not control and must decide what it can assert. Threat models need an explicit statement of who owns each control across those boundaries.13

Structural defences over filtering

The design patterns work proposes patterns that constrain agent actions so the agent cannot solve arbitrary tasks, and applies them across ten case studies of real agent applications with utility versus security trade-offs stated. CaMeL goes further by extracting control and data flow from the trusted query so untrusted data can never change program flow, and by using capabilities to block unauthorised data flows at tool call time. It solved 77% of AgentDojo tasks with provable security against 84% for the undefended system, which is the honest shape of the trade-off.56

Frameworks to structure the exercise

MITRE ATLAS is the renamed AdvML Threat Matrix and gives an interactive matrix of adversary tactics and techniques against AI plus case studies, which is useful for naming attack paths. OWASP AI Exchange provides threats, controls and a risk analysis section for narrowing to what matters in your context, and feeds ISO/IEC 27090 and EU AI Act standards work. For agents, CSA's MAESTRO is a layered agentic threat model that has been applied to real codebases.1121619

Use fictional data and authorised sandboxes. Remove employer details and secrets from any portfolio write-up. Time estimates exclude setup. Check model and cloud costs before running tests, set spending limits, and delete lab resources afterwards.

Three exercises

In order of difficulty. Free tools. Keep what you build; it is evidence.

1Threat model a retrieval augmented chatbot on one pageabout 3 h

A data flow diagram plus a threat table you can walk an engineer through in ten minutes, with each threat named in OWASP and NIST terms and each control assigned an owner.9182

  1. Draw the components using the SAIF four area split: data, infrastructure, model, application.
  2. Mark every content source as trusted or untrusted and draw the trust boundary where retrieved data enters the prompt.
  3. Walk the OWASP LLM Top 10 entries and record which apply, with the data flow each one abuses.
  4. Answer the Microsoft security review questions on training and retrieval data provenance and poisoning detection.
  5. For each threat, write one control and who owns it, separating provider controls from controls you inherit.

Tools: diagrams.net or pen and paper, OWASP Top 10 for LLM Applications, OWASP AI Exchange risk analysis section

2Prove excessive agency, then redesign the tool interfaceabout 4 h

A before and after design: a working exploitation of an over permissioned tool chain, and a revised interface that removes the capability rather than filtering the prompt.1314115

  1. Complete the PortSwigger lab on exploiting LLM APIs with excessive agency and record the exact tool call that caused impact.
  2. Complete the lab where an AI powered scanner investigates user generated content while holding another user's credentials, and note where untrusted input entered.
  3. Write the attack path in ATLAS style tactic and technique language.
  4. Redesign: list the tools, scopes and credentials the assistant actually needs, and which actions require confirmation outside the model.
  5. State the utility you lost with each restriction, following the trade-off framing in the design patterns paper.

Tools: Burp Suite Community Edition, free PortSwigger Web Security Academy account

3Threat model and harden an MCP based agentabout 6 h

A threat model of an agent plus tool server, with two implemented controls and an honest note on what the controls do not stop.151956

  1. Run the Damn Vulnerable MCP Server locally and work through its vulnerable scenarios, logging each as an attack path.
  2. Model the agent in layers, using MAESTRO's layered agentic view to place each threat at a layer.
  3. Apply one constraint pattern so the agent cannot perform arbitrary tool actions for a given task.
  4. Add a capability style check at the tool call boundary in the spirit of CaMeL, enforcing policy on data flows rather than inspecting prompt text.
  5. Re-run the scenarios and write which ones now fail, which still succeed, and what utility was lost.

Tools: Python, Docker, Damn Vulnerable MCP Server

Practice questions

Written from the concepts above, not collected from a named employer. Open one, answer it out loud, then tick the points you covered; the score stays in this browser.

Where do you draw the trust boundaries in a retrieval augmented assistant, and why there?Trust boundaries when data becomes instructions

Say your answer out loud or write it down, then tick what you covered:

0 of 4 covered

A team proposes a prompt injection classifier as their main defence for an agent with write access. What is your response?Structural defences over filtering

Say your answer out loud or write it down, then tick what you covered:

0 of 4 covered

How do you scope the blast radius of a single successful injection in an agentic system?Excessive agency and tool permissions

Say your answer out loud or write it down, then tick what you covered:

0 of 4 covered

Name a data exfiltration channel that exists even when the agent has no network tool, and the control for it.Output handling and exfiltration channels

Say your answer out loud or write it down, then tick what you covered:

0 of 4 covered

Your threat model only contains prompt injection findings. What is missing?Adversarial ML taxonomy beyond prompts

Say your answer out loud or write it down, then tick what you covered:

0 of 4 covered

How would you divide responsibility for controls between your organisation and an external model provider?Supply chain and multi actor responsibility

Say your answer out loud or write it down, then tick what you covered:

0 of 4 covered

Which frameworks would you use for an agent review, and what does each contribute?Frameworks to structure the exercise

Say your answer out loud or write it down, then tick what you covered:

0 of 4 covered

How do you present the security versus utility trade-off of a constrained agent design to product owners?Defeating Prompt Injections by Design (CaMeL)

Say your answer out loud or write it down, then tick what you covered:

0 of 4 covered

Sources

Every numbered claim above links here. P = the platform's own coding of 48 job postings.

  1. OWASP Top 10 for LLM Applications (2025 list) OWASP GenAI Security Project
  2. OWASP AI Exchange: AI Security Overview OWASP AI Exchange
  3. Guidelines for secure AI system development: Introduction UK National Cyber Security Centre
  4. NIST AI 100-2e2025: Adversarial Machine Learning, a Taxonomy and Terminology of Attacks and Mitigations NIST
  5. Design Patterns for Securing LLM Agents against Prompt Injections arXiv (Beurer-Kellner et al.)
  6. Defeating Prompt Injections by Design (CaMeL) arXiv (Debenedetti et al.)
  7. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection arXiv (Greshake, Abdelnabi et al.)
  8. Threat Modeling AI/ML Systems and Dependencies Microsoft Learn
  9. SAIF: Components of Generative AI Systems Google
  10. SAIF: Security Controls for Generative AI Systems Google
  11. Adversarial Threat Landscape for AI Systems (ATLAS announcement repository) MITRE
  12. Bing Chat: Data Exfiltration Exploit Explained Embrace The Red (Johann Rehberger)
  13. Lab: Exploiting LLM APIs with excessive agency PortSwigger Web Security Academy
  14. Lab: Exploiting AI agents to perform destructive actions PortSwigger Web Security Academy
  15. Damn Vulnerable MCP Server GitHub (harishsg993010)
  16. Agentic AI Threat Modeling Framework: MAESTRO Cloud Security Alliance
  17. AI RMF Core NIST AI Resource Center
  18. AI Risk Management Framework NIST
  19. OpenClaw Threat Model: MAESTRO Framework Analysis Cloud Security Alliance

Resources

Free first. Levels: intro means no prior knowledge of this skill; working means you can apply it on a project; advanced means research depth or specialist tooling.

Level Format

Gaps the research could not fill with a good free source: No free, vendor-neutral reference architecture for a production RAG or agent platform with control placement; the closest free items (SAIF, OWASP AI Exchange) are control catalogues, not diagrams.; Could not confirm price or hours for SANS AI security courses: the course URL redirected to a general AI training landing page with no cost shown, so no paid course is listed.; CISA/NSA joint deployment guidance for AI systems returned HTTP 404 today, so no US federal deployment checklist is included alongside the NCSC guidelines.; No free worked multi-agent threat model example from a standards body with a full data flow diagram and control decisions; available worked examples are vendor or blog write-ups.; No free instructor-led or assessed course aimed specifically at AI security architects (as opposed to red teaming or general governance).