Learn / AI red teaming and adversarial testing
AI red teaming and adversarial testing
Probing AI systems, especially LLMs, for exploitable weaknesses using adversary tactics and structured attack testing.
Researched on 2026-09-26 with AI assistance. Links and summaries can change; verify details with the original source. Not yet reviewed by a person.
What it is
AI red teaming is adversarial probing of a generative AI system to find failures an attacker could cause on purpose. The Microsoft team that red teamed over 100 generative AI products describes it as covering both security flaws and responsible AI harms, and argues the work of securing AI systems is never complete.2
The target is the whole system, not the model alone. OWASP's GenAI Red Teaming Guide organises the work into four areas: model evaluation, implementation testing, infrastructure assessment and runtime behaviour analysis.9
It is not the same as safety benchmarking. Benchmarks score a model on a fixed test set, while red teaming looks for new attacks under adversarial pressure, which is why live competitions still surface attacks that static benchmarks miss.26
Findings are usually written up against a shared vocabulary. MITRE ATLAS catalogues adversary tactics and techniques against AI systems and grew out of the Adversarial ML Threat Matrix, published by MITRE, Microsoft and 16 other organisations.110
Why postings ask for it
23 of 48 postings ask for it (48 percent), and it is universal in the two clusters that exist to break things: AI Red Team / Adversarial Testing (9 postings, 100 percent) and AI Security Research (4 postings, 100 percent).P
Builder roles ask too, because attacks now land through data an agent reads rather than through a user prompt: AI/Agent Security Engineer 44 percent of 16 postings, AI Security Architect 43 percent of 7, matching the indirect injection and agent hijacking patterns documented in the literature.P36
It is absent from AI Governance / GRC (7 postings) and Consulting (5 postings) at 0 percent, so if you sit in those clusters, treat it as background for reading red team reports rather than the skill you are hired for.P
Concepts you should be able to explain
If you can say each of these out loud in two minutes, with an example, you are ready for the technical part of an interview on this skill.
Direct injection is the user overriding instructions in their own prompt. Indirect injection puts the attacker's instructions into data the application later retrieves, such as a web page, email or repository file, so the attacker needs no interface to the model. The original paper shows this working against real systems including Bing's GPT-4 powered chat and code completion engines.3
LLM-integrated applications blur the line between data and instructions, so retrieved content can act like code the application executes. The indirect injection taxonomy covers data theft, worming, information ecosystem contamination and control over whether and how other APIs are called. When you test, ask which untrusted channels reach the prompt and what the model is allowed to do afterwards.3
When an agent processes emails, websites or code, malicious instructions in that data can derail it into exfiltrating credentials, sending phishing mail or downloading and running malware. NIST's CAISI treats agent hijacking as the same problem as indirect prompt injection and as a focus of its measurement work. Impact scales with the tools and permissions the agent holds.6
Manual jailbreaks rely on human ingenuity and are brittle. The GCG work automates the search with greedy and gradient-based techniques to produce a suffix that pushes a model towards an affirmative answer instead of a refusal. Suffixes trained on Vicuna-7B and 13B transferred to public black-box interfaces including ChatGPT, Bard and Claude.4
One of the eight lessons from red teaming 100 generative AI products is that you do not have to compute gradients to break an AI system. System-level weaknesses in application logic, retrieval paths and permissions are often cheaper routes than optimisation attacks. The same report notes that LLMs amplify existing security risks as well as introducing new ones.2
Across more than 250,000 attempts by over 400 participants against 13 frontier models, at least one successful attack was found against every model, but success rates differed sharply and did not track capability. Some universal attacks transferred across scenarios and models, and attacks found against harder-to-break models transferred to weaker ones more often than the reverse. That makes comparative red teaming useful input to model selection.6
Automation helps cover more of the risk landscape, but the human element stays central, and responsible AI harms are pervasive and hard to measure. In practice you automate breadth with scanners and keep humans for scoping, novel attack ideas and judging whether an output is actually harmful in context.2
Scanners such as garak combine static, dynamic and adaptive probes for prompt injection, data leakage, jailbreaks, toxicity and hallucination, and mark a response FAIL when a detector fires. Because garak makes several generations per prompt by default, results come as a failure rate rather than a yes or no. Detector output needs manual triage before it becomes a finding.7
Use fictional data and authorised sandboxes. Remove employer details and secrets from any portfolio write-up. Time estimates exclude setup. Check model and cloud costs before running tests, set spending limits, and delete lab resources afterwards.
Three exercises
In order of difficulty. Free tools. Keep what you build; it is evidence.
A short report of three attempted attacks on a small chat application, each mapped to an ATLAS technique and to the OWASP area it sits in.139
- Pick a local open-weight chat model and wrap it in a script with a system prompt and one fake tool.
- Try direct instruction override, then hide the same instruction in a document the script retrieves, following the indirect injection pattern.
- Record prompt, output, and what the application would have done next.
- Map each attempt to an ATLAS tactic and technique, and label it model evaluation, implementation testing, infrastructure or runtime per OWASP.
- Write one paragraph per finding: impact, preconditions, suggested control.
Tools: Python, a local open-weight model, MITRE ATLAS, OWASP GenAI Red Teaming Guide
Evidence that you can run a scanner, read the log and separate real findings from detector noise.72
- Install garak from PyPI and run garak --list_probes to see the probe families.
- Scan a small Hugging Face model with --target_type huggingface, first --spec probes.encoding then a jailbreak probe such as probes.dan.Dan_11_0.
- Read the jsonl run log and use analyse/analyse_log.py to find the probes and prompts with the most hits.
- Manually review 20 FAIL rows and mark each as true positive, false positive or unclear.
- Report failure rates per probe plus your own corrected rate, and note which detectors were unreliable.
Tools: garak, Python, Hugging Face model
A measured comparison of agent task success and attack success, with at least one adaptive attack you wrote yourself.586
- Install AgentDojo and run its baseline tasks so you see how often the agent fails with no attacker present.
- Run the bundled prompt injection attacks across the security test cases and record which security properties break.
- Write one adaptive injection tailored to a scenario the bundled attacks fail on, and test it.
- Test whether your attack still works against a second model, and note the direction of transfer.
- Optionally script the attack loop and scoring in PyRIT so the run repeats after a prompt or model change.
- Report utility under attack, attack success rate, and which defence paradigm helped most.
Tools: AgentDojo, PyRIT, Python, open-weight models
Practice questions
Written from the concepts above, not collected from a named employer. Open one, answer it out loud, then tick the points you covered; the score stays in this browser.
Explain the difference between direct and indirect prompt injection, and why the second one changes your threat model.Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
How is AI red teaming different from running a safety benchmark, and when would you do each?Red teaming vs benchmarking (s3, s7)
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
You have two days to test an internal agent that reads support tickets and can email customers. What is your plan?Agent hijacking, You do not need gradients (s3, s7)
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
What does transferability mean for adversarial prompts, and how does it affect how you test?Attack transfer (s5, s7)
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
A scanner reports a 30 percent failure rate on a jailbreak probe. What do you do before reporting it?Probes, detectors and failure rates (s8)
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
Where do automated tools help in a red team engagement, and where do they not?Automation with human judgement (s3, s8, s9)
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
How would you structure a red team report so an engineering team can act on it?s1, s10, s3
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
How do you measure whether a defence against prompt injection actually works?AgentDojo (s6)
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
Sources
Every numbered claim above links here. P = the platform's own coding of 48 job postings.
- MITRE ATLAS MITRE
- Lessons From Red Teaming 100 Generative AI Products arXiv (Microsoft AI Red Team)
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection arXiv
- Universal and Transferable Adversarial Attacks on Aligned Language Models arXiv
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents arXiv (ETH Zurich)
- Insights into AI Agent Security from a Large-Scale Red-Teaming Competition NIST CAISI
- garak, the LLM vulnerability scanner NVIDIA
- PyRIT: Python Risk Identification Tool for generative AI Microsoft
- GenAI Red Teaming Guide OWASP Gen AI Security Project
- mitre/advmlthreatmatrix: Adversarial Threat Landscape for AI Systems MITRE (GitHub)
Resources
Free first. Levels: intro means no prior knowledge of this skill; working means you can apply it on a project; advanced means research depth or specialist tooling.
- introMITRE ATLAS MITRE, Standard freeGives you the shared tactic and technique vocabulary to scope an AI red team engagement and map findings the way defenders expect.Red teamerEngineerArchitectResearcherGovernance
- introGandalf Lakera, Hands-on lab freeFirst hands-on feel for prompt injection and guardrail evasion against progressively stronger defences, with no setup required.Red teamerEngineer
- introAI Red Teaming 101 training series Microsoft, Course freeWalks a security professional through generative AI attack techniques, case studies and tooling before running their own tests.Red teamerEngineerArchitectConsultant
- introAI RMF: Generative AI Profile (NIST AI 600-1) NIST, Standard freeConnects red team findings to the risk categories and governance actions your GRC and architecture stakeholders are already tracking.GovernanceArchitectConsultantResearcher
- introGenAI Red Teaming Guide OWASP Gen AI Security Project, Standard freeLets you structure an AI red team exercise across model, implementation, infrastructure and runtime testing instead of ad hoc prompt poking.Red teamerConsultantGovernanceArchitect
- introRed Teaming LLM Applications DeepLearning.AI with Giskard, Course, about 1.5 h freeWalks you through manual and automated attacks on a chatbot in notebooks, so you can run a first assessment on your own app.EngineerRed teamerConsultant
- workingLessons From Red Teaming 100 Generative AI Products arXiv (Microsoft AI Red Team), Paper freeEight operational lessons from a working AI red team, including how to pick targets, when simple attacks beat gradient methods, and reporting.Red teamerConsultantArchitectGovernance
- workingWeb LLM attacks PortSwigger Web Security Academy, Hands-on lab freePractises indirect injection, insecure output handling and API abuse in graded web labs, the closest free match to a real pentest scope.Red teamerEngineerConsultant
- workingPyRIT Microsoft, Tool freeAutomates multi-turn attacks and scoring against your own targets, which is how you scale beyond manual prompt-by-prompt testing.Red teamerEngineerResearcher
- workinggarak NVIDIA, Tool freeRuns a broad probe suite against a model endpoint and produces a repeatable failure report you can hand to engineering.EngineerRed teamerResearcher
- workingAIRT: open-source AI red teaming course with Docker labs 0x4D31 (Adel Karimi), Hands-on lab freeEight modules with local Docker targets, so you practise attacks end to end without needing a customer environment.Red teamerEngineerResearcher
- workingNot what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection arXiv (Greshake et al.), Paper freeThe reference threat model for indirect injection, so you can reason about data-flow attack paths in RAG and tool-using systems.Red teamerEngineerArchitectResearcher
- workingBreaking Down Adversarial Machine Learning Attacks Through Red Team Challenges Olivier Laflamme, Guide, about 3 h freeWorked solutions to adversarial ML challenges, showing how model extraction and evasion attacks are actually built step by step.Red teamerResearcherEngineer
- workingThe Automation Advantage in AI Red Teaming arXiv (Dreadnode), Paper, about 2 h freeAnalyses 214,271 real attack attempts on a public challenge platform, showing which attack patterns pay off and where automation helps.Red teamerResearcherEngineer
- workingOWASP Red Teaming: A Practical Guide to Getting Started Promptfoo, Guide, about 2 h freeShows how to turn OWASP LLM risk categories into repeatable automated scans you can run in a pipeline.EngineerRed teamerConsultant
- workingPrompt Airlines AI security challenge Wiz, Hands-on lab freeFive staged challenges against a customer service chatbot, including API and chat history manipulation, not just prompt text tricks.Red teamerEngineer
- workingHackAPrompt Learn Prompting, Hands-on lab freeCompetitive tracks including a tutorial series and an indirect injection track against agents, useful for building an attack repertoire.Red teamerResearcher
- workingGenAI Red Team Lab OWASP GenAI Security Project (Red Teaming Initiative), Hands-on lab freeSelf-hosted lab that pairs with the OWASP guide, so you can rehearse the guide's test types on infrastructure you control.Red teamerEngineerConsultant
- advancedInsights into AI Agent Security from a Large-Scale Red-Teaming Competition NIST Center for AI Standards and Innovation, Guide freeFindings from a public agent attack competition, showing which attack classes actually succeed against defended agents today.ResearcherRed teamerArchitect
- advancedAgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents arXiv (Debenedetti et al.), Paper freeGives you an extensible harness plus 97 tasks and 629 security cases for measuring agent robustness and testing adaptive attacks.ResearcherRed teamerEngineer
- advancedJailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models arXiv (Chao et al.), Paper freeSets out reproducible attack success measurement, so your jailbreak results are comparable across models, runs and defences.ResearcherRed teamerEngineer
- advancedRed Teaming Language Models with Language Models arXiv (DeepMind), Paper freeThe original automated test case generation approach behind today's attacker-LLM tooling, worth reading before you build your own generator.ResearcherRed teamer
- advancedInspect UK AI Security Institute, Tool freeBuild your own attack and capability evaluations with sandboxed agents and scorers, the framework a national AI security body uses.ResearcherRed teamerEngineer
- advancedUniversal and Transferable Adversarial Attacks on Aligned Language Models arXiv (Zou et al.), Paper freeExplains the GCG optimisation attack and transfer behaviour, the basis for automated suffix attacks you will meet in tooling and papers.ResearcherRed teamer
- advancedAI Red Teamer job role path Hack The Box Academy (with Google), Course not statedTwelve modules of graded exercises covering prompt injection, data poisoning, evasion and model privacy attacks against live targets.Red teamerEngineerConsultant
Gaps the research could not fill with a good free source: No free, structured multi-week AI red teaming course with graded labs: the free options are either 1.5 hours (DeepLearning.AI) or unstructured challenge sites, and the long paths (HTB) are paid.; No free hands-on lab found today specifically for MCP and multi-agent tool chains; agent practice is mostly benchmark code (AgentDojo) rather than a guided range.; No free multimodal red teaming lab (image, audio or document-borne injection) from a primary source; the material found is papers only.; No free rules-of-engagement, scoping or report template for AI red team engagements from a standards body; the OWASP and CSA guides describe process but templates were not confirmed.; No free hands-on training on model extraction, membership inference or poisoning attacks: coverage found sits inside paid paths.
Paid options
Most of what postings ask for on this skill is covered by the free material above. These are the paid courses and certifications that touch it, with what they add and what free already covers. Showing 6 of 13: ones postings name first, then the most focused on this skill. All paid options.
- Cost
- Starting at USD 1,749 (as listed on the EXP-301 page)
- Duration
- 932h of content (level 300)
- Format
- self-paced
- Prerequisite
- Not stated on the pages we could read; OffSec describes EXP-301 as an intermediate-level exploit development course
- In the 48 postings
- Named in 1 of 48 postings: Sr. AI Red Team Engineer (listed).
Adds over free material: Builds ROP chains, DEP and ASLR bypasses and read/write primitives with grading, which matters if your AI target includes native inference runtimes or C/C++ model loaders.
Free already covers: Free exploit development series and CTF binary challenges teach the same primitives without a certification or a fixed lab set.
- Cost
- GBP 365 is listed against Red Team Ops on Zero-Point's legacy training site course list; the current course page stated no price in the text we could read
- Duration
- Study time: 20 hours (as stated on the course page)
- Format
- self-paced
- Prerequisite
- None stated; the course page lists the level as Practitioner
- Renewal
- None stated; the page states lifetime access, course updates at no extra cost, lab access with no expiry, and unlimited free exam attempts
- In the 48 postings
- Named in 1 of 48 postings: Sr. AI Red Team Engineer (listed).
Adds over free material: Hands-on Cobalt Strike adversary simulation with a licensed copy provided in the labs, plus unlimited exam retries at no extra cost.
Free already covers: Free C2 frameworks, Active Directory attack labs and public adversary emulation plans cover the concepts, but not licensed Cobalt Strike practice.
- Cost
- Starting at USD 1,749 (OffSec's Course + Cert Bundle price for a 200 or 300-level course); USD 2,749/year for Learn One
- Duration
- 671h of content; 20+ modules plus 7 challenge labs
- Format
- self-paced
- Prerequisite
- As stated: completion of PEN-200 and a passed OSCP+, or equivalent knowledge and experience
- In the 48 postings
- Named in 2 of 48 postings: Sr. AI Red Team Engineer (listed); AI Red Team Engineer for LLM Security (required, one of a list).
Adds over free material: Teaches EDR and AV evasion, custom toolchains and in-memory payload delivery against hardened enterprise targets, which free labs rarely instrument realistically.
Free already covers: Public tradecraft writeups, MITRE ATT&CK technique pages and open source loaders cover much of the theory, but not a graded hardened environment.
- Cost
- USD 1,749 once (Course + Cert Bundle) or USD 2,749/year (Learn One), per OffSec's pricing page for any 200 or 300-level course
- Duration
- not stated
- Format
- self-paced
- Prerequisite
- Not stated on the pages we could read; OffSec positions WEB-300 as an advanced white box web application course
- In the 48 postings
- Named in 1 of 48 postings: AI Red Team Engineer for LLM Security (required, one of a list).
Adds over free material: Source code review, .NET deserialization, blind SQLi and authentication bypass chains under exam conditions, the skill set you need when the LLM feature is bolted onto a web app.
Free already covers: PortSwigger Web Security Academy covers SSRF, XSS, SQLi and auth bypass labs free, and OWASP guidance covers the review method.
- Cost
- USD 1,749 once (Course + Cert Bundle, 90 days access, one exam attempt); USD 2,749/year (Learn One, one year access, two exam attempts); USD 1,699 once for the OSCP+ standalone exam
- Duration
- 321h of content; 20+ modules plus 9 challenge labs; exam is 24 hours proctored
- Format
- self-paced
- Prerequisite
- As stated: no hard prerequisite, but OffSec suggests hands-on practical knowledge of Linux and Windows administration, networking and network scripting
- Renewal
- OSCP has no expiration date; the OSCP+ designation expires 3 years from issuance
- In the 48 postings
- Named in 2 of 48 postings: Sr. AI Red Team Engineer (listed); AI Red Team Engineer for LLM Security (required, one of a list).
Adds over free material: Gives the proctored 24-hour exam and the AD and AWS challenge labs that hiring managers in the two AI red team postings treat as the entry filter.
Free already covers: Enumeration, privilege escalation and the web attacks in the syllabus are all reachable free through PortSwigger Web Security Academy, HackTricks and free TryHackMe/HTB rooms, with no exam.
- Cost
- not stated on the provider page
- Duration
- 11 modules; self-paced, sold by access duration (lab extensions in 30-day increments for Course & Cert Bundle learners)
- Format
- self-paced
- Prerequisite
- As stated: advanced level, for experienced cybersecurity practitioners, red teamers and AI professionals; solid cybersecurity fundamentals and basic familiarity with AI systems including LLMs
- Renewal
- OSAI does not expire; the OSAI+ designation expires 3 years from issuance, maintained by one of three continuing education paths. Passing OSAI+ may qualify for 40 CPE points, self-submitted to ISC2
- In the 48 postings
- Not named in any of the 48 postings.
Adds over free material: The first graded, proctored AI red team exam from a provider the two AI red team postings already name, against live LLM and agent-integrated targets.
Free already covers: OWASP LLM Top 10, prompt injection writeups and free CTF-style prompt hacking games cover most attack classes with no proctored assessment.