Learn / Python and coding for AI security work
Python and coding for AI security work
Enough Python to build a small LLM or agent app, call model APIs, write tests against them and automate attacks or checks; 67 percent of the 48 postings ask for it.
Researched on 2026-09-26 with AI assistance. Links and summaries can change; verify details with the original source. Not yet reviewed by a person.
What it is
Practical Python for AI security means being able to build the thing you are asked to test: a script that calls a model API, a small retrieval or tool-using agent, and a test harness that runs prompts and scores the answers. Most of that tooling is Python, for example Inspect installs with pip install inspect-ai and needs a provider package plus an API key in the environment.5
It also means driving the security tooling, which is Python first: garak is a scanner built around probes and detectors for LLM vulnerabilities, and PyRIT is a Python framework whose attacks, converters, scorers and memory you extend in code.36
A large part of the job is writing assertions rather than reading chat transcripts. promptfoo assertions compare model output against expected values or conditions so analysis can be automated, and Inspect splits an evaluation into a dataset, a solver and a scorer.75
The coding is defensive as well as offensive: NIST SP 800-218A extends the Secure Software Development Framework with practices and tasks specific to generative AI and dual-use foundation model development across the lifecycle, so the code you write is judged against normal secure development expectations.9
Why postings ask for it
67 percent of the 48 postings ask for coding, and the split follows how hands-on the role is: 100 percent of the 4 AI Security Research postings, 89 percent of the 9 AI Red Team postings and 81 percent of the 16 AI/Agent Security Engineer postings, against 0 percent of the 7 AI Governance / GRC postings.P
Red team and research work is written as code because the attacks are parameterised and repeated: multi-turn strategies, converters and scorers in PyRIT, or probe-and-detector runs in garak, give results you can rerun after a model or prompt change.63P
Engineering and architect postings need it because the fixes live in application code: parameterising queries fed by model output, constraining tool permissions, and sandboxing agent execution so generated code cannot reach credentials, files or the network.113P
Concepts you should be able to explain
If you can say each of these out loud in two minutes, with an example, you are ready for the technical part of an interview on this skill.
Models are reached through provider packages and an API key taken from the environment, for example pip install openai and export OPENAI_API_KEY before running an evaluation. Inspect supports over 20 providers plus local inference with HuggingFace, vLLM and SGLang, so the same test code can be pointed at a hosted or a local model. Keeping keys in the environment and making the model an argument is the base pattern for everything else.5
An Inspect task combines a dataset of labelled samples with input and target fields, a solver that produces an answer (one generate call or a full tool-using agent), and a scorer that grades the output by text comparison or model grading. Learning this shape means you can express a security test as data plus a run plus a judgement instead of a one-off prompt. It is also what makes results comparable across models.5
Model output varies between runs, so tests assert on properties and conditions rather than exact strings; promptfoo assertions compare output against expected values or conditions to automate the analysis. In security testing the assertion is usually the absence of a leaked secret, a refused action, or a well-formed and in-scope tool call. Writing the assertion first forces you to define what failure means.7
garak treats LLM security as a moving target: models give unpredictable output, are updated constantly, and what counts as a weakness in one context may not be in another, so it structures testing as probes against a target with detectors deciding whether a run failed. PyRIT adds multi-turn strategies such as Crescendo, TAP and Skeleton Key, prompt converters, true/false, Likert and classification scorers, and a SQLite memory of all conversations and results. Knowing both means you can automate attack coverage and keep the evidence.36
A ReAct agent alternates Thought, Action and Observation text, and the whole loop is just text the model reads back. The Damn Vulnerable LLM Agent shows that injecting a fake Observation and Thought hijacks the loop and makes the agent fetch another user's transactions, which is harder to stop than plain system-prompt overriding. If you cannot read and write an agent loop in Python you cannot see where these boundaries are.12
Indirect prompt injection works by placing instructions in data the application is likely to retrieve, so an attacker needs no direct interface to the model; the original work demonstrated data theft, worming and ecosystem contamination against real systems including Bing's GPT-4 powered chat and code completion engines. OWASP lists prompt injection as LLM01:2025, where user prompts alter model behaviour. In code terms, every retrieval, email body, web page or tool result is attacker-controlled input.21
OWASP LLM05:2025 Improper Output Handling covers insufficient validation and sanitisation of model output before it is used downstream. The second flag in the Damn Vulnerable LLM Agent is reached by making the model pass a UNION-based SQL injection payload into a tool argument, which is an ordinary injection bug with a model in the middle. The fix is parameterised queries, schema validation on tool arguments and encoding at the sink, all written in application code.112
OWASP LLM06:2025 Excessive Agency covers systems granted more autonomy than the task needs, and LangChain's guidance is that because you cannot predict what an agent might do its environment should be isolated so it cannot reach credentials, files or the network. Inspect ships a sandboxing system for untrusted model code on Docker, Kubernetes and similar backends. Recent work proposes design patterns for agents with provable resistance to prompt injection and discusses their utility and security trade-offs, which gives you defensible architecture arguments rather than filter-only answers.113511
Use fictional data and authorised sandboxes. Remove employer details and secrets from any portfolio write-up. Time estimates exclude setup. Check model and cloud costs before running tests, set spending limits, and delete lab resources afterwards.
Three exercises
In order of difficulty. Free tools. Keep what you build; it is evidence.
Show a Python script that calls a model and a test suite whose assertions catch a leaked secret and a malformed tool argument, with results you can rerun.75
- Write one Python function that sends a prompt to a model and returns text, taking the model name and key from environment variables.
- Put a fake secret in the system prompt and write assertions that fail if it ever appears in output, using promptfoo assertions or plain pytest.
- Add ten prompt variants as a small dataset file and loop over them so every run covers the same set.
- Rerun against a second model or temperature and note which assertions flip, then record the pass rate.
Tools: Python, pytest, promptfoo, a local model runner or a free model endpoint
Show a garak run report against a model you control plus one probe or detector you wrote yourself, and explain why each finding matters in context.34
- Install garak and run its built-in probes against a local or free-tier model, saving the report.
- Read the report and separate findings that matter for your application context from ones that do not, using the point that a weakness in one context may not be an issue in another.
- Write one custom probe reflecting a risk in your own app, for example system prompt leakage, and a detector that decides pass or fail.
- Re-run with the custom probe and write a five-line summary an engineer could act on.
Tools: Python, garak, a local model runner
Show both Damn Vulnerable LLM Agent flags obtained, a patched version, and an automated regression test that fails on the old code and passes on the new.121158
- Clone the Damn Vulnerable LLM Agent, run it in the Python virtual environment or Docker, and reach flag one by injecting a fake Observation and Thought into the ReAct loop.
- Reach flag two by forcing a UNION-based SQL injection through the GetUserTransactions tool argument.
- Patch the app: parameterise the query, take the user id from the session rather than the prompt, and validate tool arguments against a schema.
- Reduce the agent's privileges and isolate its execution environment so it cannot reach credentials or the network, citing one design pattern for your choice.
- Encode both attacks as automated tests, for example an Inspect task or a promptfoo red team run, and check that they fail before the patch and pass after.
Tools: Python, Docker, Damn Vulnerable LLM Agent, Inspect, promptfoo, a local model runner
Practice questions
Written from the concepts above, not collected from a named employer. Open one, answer it out loud, then tick the points you covered; the score stays in this browser.
Your test prompt gets a different answer every run. How do you write a security test you can trust?Assertions instead of eyeballing
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
Walk me through the structure of an evaluation you would write for a customer support agent.Dataset, solver, scorer
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
What is the difference between direct and indirect prompt injection, and what does that change in your test code?Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
An agent uses a ReAct loop. Where would you attack it and why is that harder to filter than a system-prompt override?The agent loop and tool calls
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
A developer says the model is safe because it refuses harmful requests. What else do you check in the code?Model output is untrusted input
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
How would you reduce the blast radius of an agent that can browse and run code?Least privilege, sandboxing and design patterns
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
You have a garak report with dozens of failures. How do you turn it into work an engineering team will do?Probes, detectors and scanners
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
How do you keep these tests useful after the model is upgraded next month?promptfoo docs: LLM red teaming guide (open source)
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
Sources
Every numbered claim above links here. P = the platform's own coding of 48 job postings.
- OWASP Top 10 for LLM and Gen AI Applications (2025) OWASP Gen AI Security Project
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection arXiv (Greshake et al.)
- garak: A Framework for Security Probing Large Language Models arXiv
- NVIDIA/garak, the LLM vulnerability scanner NVIDIA (GitHub)
- Inspect: an open-source framework for large language model evaluations UK AI Security Institute
- PyRIT, Python Risk Identification Tool documentation Microsoft
- promptfoo docs: Assertions and metrics promptfoo
- promptfoo docs: LLM red teaming guide (open source) promptfoo
- NIST SP 800-218A, Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile NIST
- Secure Software Development Framework project page NIST
- Design Patterns for Securing LLM Agents against Prompt Injections arXiv
- Damn Vulnerable LLM Agent Reversec Labs (formerly WithSecure Labs)
- LangChain security policy and agent sandboxes LangChain
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents arXiv
- Web LLM attacks learning path and labs PortSwigger Web Security Academy
Resources
Free first. Levels: intro means no prior knowledge of this skill; working means you can apply it on a project; advanced means research depth or specialist tooling.
- introAutomate the Boring Stuff with Python Al Sweigart, Book freeGets a non-developer writing working Python scripts for files, HTTP requests and parsing, the base layer for any attack or check automation.EngineerRed teamerConsultantArchitect
- introHugging Face LLM Course, chapter 1 Hugging Face, Course freeTeaches you to load, run and fine-tune models in Python so you can test real models instead of only hosted APIs.EngineerRed teamerResearcher
- introLangChain security guidance LangChain, Guide, about 1 h freeShows where a Python LLM app leaks authority (tools, credentials, file and shell access) so you can review your own scaffolding code.EngineerArchitectConsultant
- introNIST Secure Software Development Framework (SSDF) project NIST, Standard freeGives the baseline secure development practices your Python and model code must meet, and the vocabulary reviewers and auditors will use.EngineerArchitectGovernanceConsultant
- introCS50's Introduction to Programming with Python (CS50P) Harvard University, Course freeGets you from no Python to writing functions, classes, exceptions, regexes and unit tests, the base every AI security script needs.EngineerRed teamerArchitectConsultantResearcher
- introOWASP Top 10 for LLM and Gen AI Applications (2025) OWASP Gen AI Security Project, Standard, about 3 h freeGives you the shared vocabulary (improper output handling, excessive agency, unbounded consumption) to name what your code should test for.EngineerRed teamerArchitectGovernanceConsultant
- introLakera Agent Breaker challenges Lakera, Hands-on lab, about 2 h freeBrowser challenges that build intuition for how prompt-level attacks land, before you automate the same ideas in Python.Red teamerEngineerConsultant
- workingPyRIT, Python Risk Identification Tool for generative AI Microsoft, Tool, about 8 h freeTeaches you to script multi-turn attacks (Crescendo, TAP, Skeleton Key) and scan targets from Python rather than by hand.Red teamerEngineerResearcherConsultant
- workingInspect: evaluation framework for LLMs UK AI Security Institute and Meridian Labs, Tool freeTeaches the task, solver and scorer pattern in Python so you can write security evals that run in CI against any model.EngineerResearcherRed teamer
- workingpromptfoo promptfoo (open source), Tool freeTurns prompt and jailbreak testing into config plus code you can run per commit, which is how AppSec teams already work.EngineerRed teamerConsultant
- workingAI Red Teaming Playground Labs Microsoft (Minnich, Lopez, Pouliot), Hands-on lab freeSelf-host the Black Hat USA 2024 AI red teaming challenges with Docker or Kubernetes and practise attacks against targets you control.Red teamerEngineerConsultant
- workingDamn Vulnerable LLM Agent WithSecure Labs (now Reversec), Hands-on lab freeA small ReAct agent you run locally to practise thought and tool-call injection, then read the Python that made it exploitable.Red teamerEngineer
- workingHugging Face AI Agents Course Hugging Face, Course freeWalks you through coding agents with tools and memory, so you can reproduce the tool-abuse and memory-poisoning paths you need to test.EngineerRed teamerArchitectResearcher
- workingDeepTeam Confident AI (open source), Tool freeA maintained Python library for scripted red team runs against LLMs and agents, useful when you need attack coverage as code.Red teamerEngineerConsultant
- workinggarak: A Framework for Security Probing Large Language Models arXiv (NVIDIA authors), Paper, about 2 h freeExplains the probe, generator and detector model behind the scanner you run, so you can extend it instead of only using defaults.Red teamerResearcherEngineer
- workingNIST SP 800-218A, Secure Software Development Practices for Generative AI (SSDF Community Profile) NIST, Standard, about 4 h freeMaps AI-specific development tasks onto SSDF practices, so your code, tests and pipeline evidence line up with what auditors expect.EngineerArchitectGovernanceConsultant
- workinggarak, the LLM vulnerability scanner NVIDIA, Tool, about 5 h freeYou learn to run probe and detector suites against a model from the CLI, and to write your own probes in Python.Red teamerEngineerResearcher
- workingWeb LLM attacks learning path and labs PortSwigger Web Security Academy, Hands-on lab, about 8 h freeGraded labs where you map LLM API attack surface and chain injection into command injection, XSS and destructive agent actions.Red teamerEngineerConsultant
- workingOpenAI Cookbook OpenAI, Guide freeRunnable notebooks for API calls, tool use, agents and evaluation, useful as the starting code for your own attack and check scripts.EngineerRed teamerConsultantResearcher
- workingNot what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection arXiv (Greshake et al.), Paper, about 3 h freeThe foundational indirect injection paper: gives you the attack taxonomy and payload patterns to encode in your own test harness.Red teamerEngineerArchitectResearcher
- advancedAgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents arXiv (ETH Zurich SPY Lab), Paper, about 3 h freeShows how to code an agent benchmark with tasks, injections and utility scoring, the pattern most later defence evaluations copy.ResearcherRed teamerEngineer
- advancedDesign Patterns for Securing LLM Agents against Prompt Injections arXiv (Beurer-Kellner et al.), Paper, about 3 h freeGives implementable patterns (plan-then-execute, dual model, action selector) you can code into an agent to limit injection impact.EngineerArchitectResearcherConsultant
- advancedSecAlign: Defending Against Prompt Injection with Preference Optimization arXiv (Chen et al.), Paper, about 3 h freeShows how a defence is trained and measured, giving you the evaluation setup to judge vendor claims about injection-resistant models.ResearcherEngineerRed teamer
- advancedBuild a Large Language Model (From Scratch), code repository Sebastian Raschka, Book free (code; the book is paid)Notebooks that build tokenisation, attention, pretraining and fine-tuning in PyTorch, so model-level attacks stop being a black box.ResearcherEngineerRed teamer
Gaps the research could not fill with a good free source: No free course found today that teaches secure coding specifically for LLM and agent applications with graded exercises; the free material is either general Python or attack labs.; Could not confirm hours or price from the provider page for the Hugging Face AI Agents Course (index snippet says free, page not fetched).; No free, well-maintained lab found for writing pytest-style regression tests and CI gates against model APIs beyond the promptfoo docs, so that is a single point of dependence.; No free primary-source tutorial found for writing and hardening MCP servers in Python; the vulnerable-agent labs available are small community repos.; Nothing found that is aimed at GRC readers who need only enough Python to read and review AI test code (that cluster asks for coding in 0 percent of postings, so the gap may not matter).
Paid options
Most of what postings ask for on this skill is covered by the free material above. These are the paid courses and certifications that touch it, with what they add and what free already covers. Showing 6 of 8: ones postings name first, then the most focused on this skill. All paid options.
- Cost
- Starting at USD 1,749 (as listed on the EXP-301 page)
- Duration
- 932h of content (level 300)
- Format
- self-paced
- Prerequisite
- Not stated on the pages we could read; OffSec describes EXP-301 as an intermediate-level exploit development course
- In the 48 postings
- Named in 1 of 48 postings: Sr. AI Red Team Engineer (listed).
Adds over free material: Builds ROP chains, DEP and ASLR bypasses and read/write primitives with grading, which matters if your AI target includes native inference runtimes or C/C++ model loaders.
Free already covers: Free exploit development series and CTF binary challenges teach the same primitives without a certification or a fixed lab set.
- Cost
- Starting at USD 1,749 (OffSec's Course + Cert Bundle price for a 200 or 300-level course); USD 2,749/year for Learn One
- Duration
- 671h of content; 20+ modules plus 7 challenge labs
- Format
- self-paced
- Prerequisite
- As stated: completion of PEN-200 and a passed OSCP+, or equivalent knowledge and experience
- In the 48 postings
- Named in 2 of 48 postings: Sr. AI Red Team Engineer (listed); AI Red Team Engineer for LLM Security (required, one of a list).
Adds over free material: Teaches EDR and AV evasion, custom toolchains and in-memory payload delivery against hardened enterprise targets, which free labs rarely instrument realistically.
Free already covers: Public tradecraft writeups, MITRE ATT&CK technique pages and open source loaders cover much of the theory, but not a graded hardened environment.
- Cost
- USD 1,749 once (Course + Cert Bundle) or USD 2,749/year (Learn One), per OffSec's pricing page for any 200 or 300-level course
- Duration
- not stated
- Format
- self-paced
- Prerequisite
- Not stated on the pages we could read; OffSec positions WEB-300 as an advanced white box web application course
- In the 48 postings
- Named in 1 of 48 postings: AI Red Team Engineer for LLM Security (required, one of a list).
Adds over free material: Source code review, .NET deserialization, blind SQLi and authentication bypass chains under exam conditions, the skill set you need when the LLM feature is bolted onto a web app.
Free already covers: PortSwigger Web Security Academy covers SSRF, XSS, SQLi and auth bypass labs free, and OWASP guidance covers the review method.
- Cost
- USD 1,749 once (Course + Cert Bundle, 90 days access, one exam attempt); USD 2,749/year (Learn One, one year access, two exam attempts); USD 1,699 once for the OSCP+ standalone exam
- Duration
- 321h of content; 20+ modules plus 9 challenge labs; exam is 24 hours proctored
- Format
- self-paced
- Prerequisite
- As stated: no hard prerequisite, but OffSec suggests hands-on practical knowledge of Linux and Windows administration, networking and network scripting
- Renewal
- OSCP has no expiration date; the OSCP+ designation expires 3 years from issuance
- In the 48 postings
- Named in 2 of 48 postings: Sr. AI Red Team Engineer (listed); AI Red Team Engineer for LLM Security (required, one of a list).
Adds over free material: Gives the proctored 24-hour exam and the AD and AWS challenge labs that hiring managers in the two AI red team postings treat as the entry filter.
Free already covers: Enumeration, privilege escalation and the web attacks in the syllabus are all reachable free through PortSwigger Web Security Academy, HackTricks and free TryHackMe/HTB rooms, with no exam.
- Cost
- not stated on the provider page
- Duration
- 3 days instructor-led or 18 hours self-paced, 18 CPEs
- Format
- mixed
- Prerequisite
- not stated
- In the 48 postings
- Not named in any of the 48 postings.
Adds over free material: Structured practice in using AI tooling as part of an offensive engagement, with the GIAC Offensive AI Analyst (GOAA) exam as the proof point.
Free already covers: Open-source offensive AI tooling, conference talks and the MITRE ATLAS case studies cover most of the individual techniques without a fee.
- Cost
- not stated on the provider page
- Duration
- 11 modules, self-paced; sold via Course and Cert Exam Bundle, Learn One or Learn Enterprise from 31 March 2026; lab extensions in 30-day increments for bundle learners
- Format
- self-paced
- Prerequisite
- advanced level: solid cybersecurity fundamentals and basic familiarity with AI systems including LLMs; aimed at experienced practitioners, red teamers and AI professionals
- Renewal
- OSAI does not expire; the OSAI+ designation expires 3 years from issuance and is maintained through one of three continuing education paths
- In the 48 postings
- Not named in any of the 48 postings.
Adds over free material: A proctored offensive exam against AI-integrated environments from the vendor whose other certs the red-team postings do ask for, worth 40 ISC2 CPE points on a pass.
Free already covers: Microsoft's free AI Red Teaming material and PyRIT, plus OWASP LLM guidance and public jailbreak research, teach the techniques; what you cannot get free is the graded exam.