Learn / Application security for LLM-backed software
Application security for LLM-backed software
Applying secure design, review and testing practices to applications that embed LLMs, agents or RAG pipelines.
Researched on 2026-09-26 with AI assistance. Links and summaries can change; verify details with the original source. Not yet reviewed by a person.
What it is
Application security for LLM-backed software means treating the model, its prompts, its retrieved context and its tools as part of the application's attack surface. PortSwigger's working method is to enumerate direct and indirect inputs, work out what data and APIs the model can reach, then probe that surface.2
The risks are catalogued: the OWASP Top 10 for LLM Applications 2025 lists prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption.1
It is ordinary secure design plus one new property: LLM-integrated applications blur the line between data and instructions, so content the application merely retrieves can act like code that redirects the application's functionality and API calls.4
Standards bodies frame it as a lifecycle problem rather than a model problem. NCSC and CISA guidance says security must be a core requirement across design, development, deployment and operation of AI systems, alongside existing cyber security practice.3
Why postings ask for it
14 of 48 postings (29 percent) ask for it, and it is concentrated where software gets designed: 57 percent of AI Security Architect postings and 31 percent of AI/Agent Security Engineer postings.P
Architect and engineer roles have to decide what tools an agent may call and what happens to model output, because excessive agency and improper output handling are design decisions, not model settings.P12
GRC postings ask at 29 percent because assurance work now means checking secure-by-design claims across the AI lifecycle, not just reviewing a model card.P3
Concepts you should be able to explain
If you can say each of these out loud in two minutes, with an example, you are ready for the technical part of an interview on this skill.
LLMs do not reliably separate operator instructions from content they are given to process; everything is concatenated into one token sequence. Greshake and co-authors showed that processing retrieved prompts can act like arbitrary code execution, changing application behaviour and controlling whether other APIs are called. Assume any text or image reaching the model can carry instructions.46
The attacker never touches the chat box; they plant instructions in data the application is likely to retrieve, such as a web page, email, ticket or document. The original paper demonstrated this against real systems including Bing's GPT-4 powered chat and code-completion engines, with impacts including data theft and worming. This is the attack class that breaks RAG pipelines and agents.4
A design becomes exploitable when one system combines access to private data, exposure to untrusted content, and the ability to communicate externally. Any outbound HTTP request, image load or clickable link can be the exfiltration path. Vendors usually fixed reported cases by closing the exfiltration vector, so removing one leg of the trifecta is a real mitigation.6
Excessive agency is when the model can reach APIs that touch sensitive data or take actions, and can be talked into using them outside intended scope. Mapping starts by asking the model which tools it has, then testing each one. A confirmation step before the model calls an external API is a basic control, since users often do not know a call happened.21
Model output is untrusted input to whatever consumes it: a browser, a shell, a database, another service. PortSwigger notes that attacking an LLM integration resembles SSRF, since you abuse a server-side component to reach systems you cannot touch directly, so classic web exploits such as path traversal and SQL injection should be sent through the model to every API it can call.21
Retrieval adds its own risk classes: vector and embedding weaknesses, plus poisoning of embedding or fine-tuning data. In review terms that means asking who can write to the corpus, whether tenant documents are isolated in the index, and whether retrieved chunks are labelled as untrusted data before they reach the prompt.1
Because filtering is probabilistic, Beurer-Kellner, Tramer and colleagues argue for design patterns that constrain what an agent can do after it has seen untrusted content, and analyse the utility cost of each pattern. In a design review this shifts the question from how good is the guardrail to what damage remains possible if the injection succeeds.76
AgentDojo provides an extensible environment with 97 realistic tasks and 629 security test cases, and its authors found that existing injection attacks break some security properties but not all. Open scanners such as garak and promptfoo generate adversarial test suites you can rerun in CI, so injection testing becomes a regression suite rather than a one-off assessment.589
Use fictional data and authorised sandboxes. Remove employer details and secrets from any portfolio write-up. Time estimates exclude setup. Check model and cloud costs before running tests, set spending limits, and delete lab resources afterwards.
Three exercises
In order of difficulty. Free tools. Keep what you build; it is evidence.
You can walk an interviewer through inputs, reachable data, reachable APIs and the exploit path, using vocabulary from a recognised methodology.21
- Read the Web LLM attacks topic and write down the three-step method: identify direct and indirect inputs, list data and APIs the model can reach, probe that surface.
- Work the Web Security Academy LLM labs, including the indirect injection lab against an AI-powered scanner.
- For each lab, record which OWASP LLM Top 10 entry it maps to.
- Write a one-page surface map for one LLM feature at your own employer or an open source app, working from design documents and code you already have access to; live probing of an employer's system needs written authorisation.
Tools: PortSwigger Web Security Academy (free labs), Burp Suite Community
You can produce a scan report against a local model or an endpoint you own, triage false positives, and rerun it as a regression check.8916
- Install garak and run its probes against a locally hosted model to get a baseline report.
- Run promptfoo's red team quickstart against the same target so you have two independent test sets.
- Triage 10 findings by hand: mark real, unclear or false, and note why.
- Map surviving findings to OWASP LLM Top 10 entries and to the exfiltration path they would need to cause harm.
- Save the config so a code change can be rescanned.
Tools: garak, promptfoo (open source), a locally hosted open-weights model
You can present a before-and-after architecture that limits blast radius after a successful injection, with the utility cost stated honestly.756103
- Pick an agent design with private data access, untrusted content exposure and an outbound channel, and document the trifecta explicitly.
- Read the design patterns paper and choose two patterns that constrain the agent after it reads untrusted data.
- Redraw the design with those patterns and state what the agent can no longer do for the user.
- Stand up AgentDojo locally and run its security test cases against your original and revised tool policies.
- Write the result as a short review memo referencing the relevant OWASP agentic risks and NCSC secure-by-design expectations.
Tools: AgentDojo (code released with the paper), promptfoo, a locally hosted open-weights model
Practice questions
Written from the concepts above, not collected from a named employer. Open one, answer it out loud, then tick the points you covered; the score stays in this browser.
Why can't you fix prompt injection by telling the model to ignore instructions found in retrieved content?Instruction and data confusion
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
How would you scope an assessment of a customer support assistant that has order and stock APIs?Web LLM attacks
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
Explain the lethal trifecta and use it to triage two agent designs.The lethal trifecta
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
A developer renders model output directly in the browser and passes part of it to a shell command. What do you tell them?Improper output handling
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
What extra review questions does a RAG pipeline add over a plain chat feature?RAG and embedding weaknesses
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
How would you prove to an auditor that injection testing is part of the delivery pipeline, not a one-off?Testing and regression harnesses
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
Where do you draw the trust boundaries in an agent that reads email and can send HTTP requests?Design patterns instead of prompt pleading
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
Which risk catalogue would you use to structure a report on an agentic application, and why?OWASP Top 10 for LLM Applications 2025
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
Sources
Every numbered claim above links here. P = the platform's own coding of 48 job postings.
- OWASP Top 10 for LLM Applications 2025 OWASP GenAI Security Project
- Web LLM attacks PortSwigger Web Security Academy
- Guidelines for secure AI system development: Introduction UK National Cyber Security Centre
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection arXiv (Greshake et al.)
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents arXiv (Debenedetti et al.)
- The lethal trifecta for AI agents: private data, untrusted content, and external communication Simon Willison
- Design Patterns for Securing LLM Agents against Prompt Injections arXiv (Beurer-Kellner, Tramer et al.)
- garak, the LLM vulnerability scanner NVIDIA
- Red team: getting started Promptfoo
- OWASP Top 10 for Agentic Applications for 2026 OWASP GenAI Security Project
- MITRE ATLAS MITRE
Resources
Free first. Levels: intro means no prior knowledge of this skill; working means you can apply it on a project; advanced means research depth or specialist tooling.
- introOWASP Top 10 for LLM Applications 2025 OWASP GenAI Security Project, Standard freeGives you the shared vocabulary and risk list that review checklists, test plans and findings for LLM applications are written against.EngineerRed teamerArchitectGovernanceConsultantResearcher
- introWeb LLM attacks learning path PortSwigger Web Security Academy, Course freeTakes you from zero to mapping LLM attack surface and exploiting real chatbot targets with Burp, using the AppSec workflow you already know.EngineerRed teamerConsultant
- introRed Teaming LLM Applications DeepLearning.AI with Giskard, Course, about 1.5 h freeShort notebook course that takes you from manual prompt-injection probing to automated vulnerability scans of a chatbot you can rerun on your own app.EngineerRed teamerConsultant
- introLakera Agent Breaker (Gandalf) Lakera, Hands-on lab freeBrowser challenges where you break guarded LLM apps level by level, building intuition for why input filtering alone fails.Red teamerEngineerConsultant
- introLLM-OWASP-LAB: vulnerable AI apps for the LLM Top 10 Ritesh Sahu (community project), Hands-on lab freeSelf-hosted vulnerable apps let you reproduce each OWASP LLM Top 10 category locally before you test anything you are paid to test.EngineerRed teamer
- introWeb LLM attacks (Web Security Academy topic and labs) PortSwigger, Hands-on lab freeYou exploit real LLM-backed web apps through prompt injection, insecure tool APIs and poisoned data, then know what to test for in review.EngineerRed teamerConsultant
- introHackAPrompt 2.0 Learn Prompting, Hands-on lab freeCompetitive prompt-hacking tracks, including an indirect injection track against agents, build the attacker intuition you need before reviewing prompts and tool wiring.Red teamerEngineerResearcher
- introOWASP Agentic Skills Top 10 (public review v1) OWASP Foundation, Guide freeLists the risks in agent skill and manifest files (SKILL.md, skill.json, manifest.json) so you can review what developers drop into coding agents.EngineerArchitectConsultantGovernance
- workingOWASP Top 10 for Agentic Applications for 2026 OWASP GenAI Security Project, Standard freeExtends the LLM risk list to agents that plan and call tools, which is what you actually review once an app gains memory and autonomy.ArchitectEngineerRed teamerGovernanceConsultant
- workingNIST AI 600-1: Generative AI Profile (AI RMF) NIST, Standard freeLets you tie application level controls to twelve named generative AI risks and the AI RMF functions auditors and customers ask about.GovernanceArchitectConsultant
- workingMITRE ATLAS MITRE, Standard freeGives tactic and technique IDs for AI attacks so your test plans and reports line up with the taxonomy defenders map detections to.Red teamerArchitectEngineerResearcher
- workingOWASP AI Exchange OWASP, Guide freeThreat and control reference you can use as a review checklist across input, development time and runtime, with links into AI Act and ISO work.ArchitectEngineerGovernanceConsultant
- workingPrompt injection: 163 posts and running Simon Willison, Guide freeRunning archive of real injection incidents and failed mitigations, useful when you need evidence that a proposed filter will not hold.EngineerRed teamerArchitectConsultant
- workingLab: Exploiting AI agents to perform destructive actions PortSwigger Web Security Academy, Hands-on lab freeYou plant instructions in user-generated content and make an authenticated AI scanner act on them, the core indirect injection case in agent reviews.Red teamerEngineer
- workingDamn Vulnerable LLM Agent Reversec Labs, Hands-on lab freeA deliberately broken ReAct agent where you abuse thought and tool traces, the fastest way to see how agent plumbing leaks and misbehaves.Red teamerEngineerConsultant
- workinggarak, the LLM vulnerability scanner NVIDIA, Tool freeRun repeatable probe suites (injection, jailbreak, leakage, toxicity) against a model or endpoint and attach the output to an AppSec test report.EngineerRed teamerConsultant
- workingPromptfoo LLM red teaming guide and scanner Promptfoo, Tool freeShows how to generate adversarial inputs and wire the results into CI, and separates model layer from application layer test goals.EngineerRed teamerArchitect
- workingNot what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection arXiv (Greshake et al.), Paper freeThe paper behind OWASP LLM01 and most injection guidance: shows why retrieved content is untrusted input in any RAG or tool-using app.EngineerRed teamerArchitectResearcher
- advancedAgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents Debenedetti et al. (arXiv), Paper freeThe benchmark most agent defence claims are measured against; use it to test your own agent under attack instead of trusting vendor numbers.ResearcherRed teamerEngineer
- advancedIndirect Prompt Injections: Are Firewalls All You Need, or Stronger Benchmarks? arXiv, Paper freeShows a simple agent-tool interface firewall scoring near perfect on four public benchmarks, which tells you how much benchmark results are worth.ArchitectResearcherRed teamer
- advancedSecuring Agentic Applications Guide 1.0 OWASP GenAI Security Project, Agentic Security Initiative, Guide freeDesign and deployment controls for agentic systems: identity, tool mediation, memory, human approval, written for people building not just assessing.ArchitectEngineerGovernanceConsultant
- advancedMELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents arXiv, Paper freeExplains a masked re-execution and tool-comparison defence, so you can judge vendor injection-defence claims against a specific published mechanism.ResearcherArchitectEngineer
- advancedAgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents arXiv, Paper freeMethod for automatically searching injection payloads against a black-box agent, useful when you need coverage beyond hand-written test prompts.Red teamerResearcherEngineer
- advancedDVAR, Damn Vulnerable Agentic RAG 0xsu3ks (community project), Hands-on lab freePractise poisoning a retrieval corpus and pivoting through an agentic RAG app, the failure path most enterprise LLM projects actually build.Red teamerEngineerArchitect
- advancedSEC545: GenAI and LLM Application Security SANS Institute, Course not statedInstructor-led labs on securing agentic systems, AI integrations and MLSecOps pipelines, with a certification employers can check.EngineerArchitectConsultant
Gaps the research could not fill with a good free source: No free instructor-led or long-form course equivalent to SANS SEC545 for secure design review of LLM applications; free options are short notebook courses.; No confirmed price or duration for SANS SEC545 on the provider page.; No free, vendor-neutral threat modelling worksheet or STRIDE-style template specifically for RAG and agent architectures from a standards body.; No free mapping from LLM AppSec controls to ISO/IEC 42001 or SOC 2 evidence that GRC leads could reuse.; Free RAG-specific hands-on ranges are all small community repos with few stars; no well-maintained RAG lab from a major provider.; No free book-length treatment of LLM application security found in today's searches.
Paid options
Most of what postings ask for on this skill is covered by the free material above. These are the paid courses and certifications that touch it, with what they add and what free already covers. Showing 6 of 8: ones postings name first, then the most focused on this skill. All paid options.
- Cost
- Starting at USD 1,749 (OffSec's Course + Cert Bundle price for a 200 or 300-level course); USD 2,749/year for Learn One
- Duration
- 671h of content; 20+ modules plus 7 challenge labs
- Format
- self-paced
- Prerequisite
- As stated: completion of PEN-200 and a passed OSCP+, or equivalent knowledge and experience
- In the 48 postings
- Named in 2 of 48 postings: Sr. AI Red Team Engineer (listed); AI Red Team Engineer for LLM Security (required, one of a list).
Adds over free material: Teaches EDR and AV evasion, custom toolchains and in-memory payload delivery against hardened enterprise targets, which free labs rarely instrument realistically.
Free already covers: Public tradecraft writeups, MITRE ATT&CK technique pages and open source loaders cover much of the theory, but not a graded hardened environment.
- Cost
- USD 1,749 once (Course + Cert Bundle) or USD 2,749/year (Learn One), per OffSec's pricing page for any 200 or 300-level course
- Duration
- not stated
- Format
- self-paced
- Prerequisite
- Not stated on the pages we could read; OffSec positions WEB-300 as an advanced white box web application course
- In the 48 postings
- Named in 1 of 48 postings: AI Red Team Engineer for LLM Security (required, one of a list).
Adds over free material: Source code review, .NET deserialization, blind SQLi and authentication bypass chains under exam conditions, the skill set you need when the LLM feature is bolted onto a web app.
Free already covers: PortSwigger Web Security Academy covers SSRF, XSS, SQLi and auth bypass labs free, and OWASP guidance covers the review method.
- Cost
- USD 1,749 once (Course + Cert Bundle, 90 days access, one exam attempt); USD 2,749/year (Learn One, one year access, two exam attempts); USD 1,699 once for the OSCP+ standalone exam
- Duration
- 321h of content; 20+ modules plus 9 challenge labs; exam is 24 hours proctored
- Format
- self-paced
- Prerequisite
- As stated: no hard prerequisite, but OffSec suggests hands-on practical knowledge of Linux and Windows administration, networking and network scripting
- Renewal
- OSCP has no expiration date; the OSCP+ designation expires 3 years from issuance
- In the 48 postings
- Named in 2 of 48 postings: Sr. AI Red Team Engineer (listed); AI Red Team Engineer for LLM Security (required, one of a list).
Adds over free material: Gives the proctored 24-hour exam and the AD and AWS challenge labs that hiring managers in the two AI red team postings treat as the entry filter.
Free already covers: Enumeration, privilege escalation and the web attacks in the syllabus are all reachable free through PortSwigger Web Security Academy, HackTricks and free TryHackMe/HTB rooms, with no exam.
- Cost
- USD 749 (US and all regions not otherwise listed, and Asia Pacific, Middle East, Africa); EUR 719.04 in EMEA; GBP 606.69 in the UK. Reschedule USD 50, cancellation USD 100.
- Duration
- not stated
- Format
- exam only
- Prerequisite
- 5 years required work experience, stated on the certification page. Exam-only purchase must be scheduled and sat within 365 days.
- In the 48 postings
- Named in 3 of 48 postings: Cyber - AI Security - Senior - Consulting (a plus); Information Security Architect - AI (listed); Security Engineer, AI (a plus).
Adds over free material: Covers the eight domains an AI security architect is expected to already know, including IAM, security architecture and engineering, and software development security, and is the credential most often listed by name in the GRC and architect postings here.
Free already covers: Nothing free replaces the credential itself, but the underlying material (NIST SP 800 series, cloud provider architecture guidance, OWASP) is all free.
- Cost
- not stated on the provider page
- Duration
- 30+ guided exercises, 60 days browser-based lab access, 36 CPE points, one exam attempt included
- Format
- self-paced
- Prerequisite
- not stated
- In the 48 postings
- Not named in any of the 48 postings.
Adds over free material: One of the few paid exams scoped to Model Context Protocol specifically: attacking, assessing and hardening MCP servers including tool poisoning, prompt injection, supply chain and agentic defences.
Free already covers: The MCP specification, OWASP agentic security guidance and public MCP tool-poisoning research already describe these attack patterns, and you can run a vulnerable MCP server locally for free.
- Cost
- not stated on the provider page
- Duration
- 7 chapters, 30+ guided exercises, 60 days browser-based lab access, 36 CPE points, one exam attempt included
- Format
- self-paced
- Prerequisite
- basic Linux command line (ls, cd, mkdir); familiarity with Python, Go or Ruby helps but is not required
- In the 48 postings
- Not named in any of the 48 postings.
Adds over free material: Hands-on labs that pair the OWASP LLM Top 10 and MITRE ATLAS tactics with things you build and break yourself (chatbot, fine-tuned model, RAG system, TextAttack and BackdoorBox exercises) plus an exam and lifetime instructor support channel.
Free already covers: The OWASP LLM Top 10 and MITRE ATLAS are free and already give the taxonomy, mitigations and real incident write-ups this syllabus is organised around.