Learn / Detection, logging and response for AI systems
Detection, logging and response for AI systems
Knowing what an attack on an LLM or agent looks like in logs, which signals to collect, and how to respond when a model or agent misbehaves.
Researched on 2026-09-26 with AI assistance. Links and summaries can change; verify details with the original source. Not yet reviewed by a person.
What it is
Detection and response for AI systems means collecting telemetry from model calls, agent tool calls and the hosts that run them, then deciding what normal looks like and what to do when it is not. The UK NCSC puts this in the operation and maintenance stage of the AI lifecycle: measure model and system outputs so you can see sudden and gradual behaviour changes affecting security, and log inputs such as inference requests, queries and prompts so you can audit, investigate and remediate after misuse or compromise.4
It is partly a new data problem. OpenTelemetry defines GenAI span conventions with fields such as gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.agent.id and error.type, with prompts and system instructions treated as opt-in rather than default content; the conventions are still marked Status: Development and have moved to a dedicated repository.23
It is also an old detection problem with a new parent process. In the July 2026 Hugging Face intrusion, an agent-driven chain started with untrusted dataset content abusing a config-driven loader in a production Kubernetes worker pod (local file disclosure via an HDF5 external raw-storage page, then Jinja2 template injection for code execution), then credential harvest and multi-cluster lateral movement, behaviours that existing endpoint and SIEM rules already watch.6
The response side needs its own runbook. OWASP's GenAI Security Project commissioned the GenAI Incident Response Guide 1.0 (published 28 July 2025) for security practitioners generally, explicitly not assuming deep GenAI knowledge, and NIST published AI 800-4 on the challenges of monitoring deployed AI systems in March 2026.189
Why postings ask for it
Across 48 postings, 4 ask for this skill (8%), so it is not yet a mainstream ask; it is concentrated where someone has to specify and operate the pipeline rather than break the model.P
29% of the 7 AI Security Architect postings ask for it, against 12% of 16 AI/Agent Security Engineer postings, which matches the work: architects decide which prompt, tool-call and host signals get collected and retained, engineers wire the traces and guardrail events into a pipeline the SOC can query.P24
Research, red team, GRC and consulting clusters ask at 0%, so treat this as a transferable angle for engineer and architect applications: the Hugging Face chain shows conventional detections still do much of the work.P6
Concepts you should be able to explain
If you can say each of these out loud in two minutes, with an example, you are ready for the technical part of an interview on this skill.
OpenTelemetry's GenAI conventions model a model call as a span with required attributes gen_ai.operation.name and gen_ai.provider.name, plus conditionally required fields like gen_ai.request.model, gen_ai.agent.id, gen_ai.agent.name, gen_ai.agent.version, server.address and error.type. Full system instructions are Opt-In, not Recommended, so prompt content is a deliberate collection decision. The conventions are Status: Development and moved out of the main semantic-conventions repository, so pin a version before building detections on field names.23
NCSC tells providers to monitor and log inputs such as inference requests, queries and prompts in line with privacy and data protection requirements, so that compliance, audit, investigation and remediation are possible after compromise or misuse. That includes explicit detection of out-of-distribution or adversarial inputs, including inputs aimed at data preparation steps such as image cropping and resizing. In practice you negotiate retention, redaction and access controls before you get prompt text into the SIEM.4
NCSC's operation and maintenance guidance asks you to measure model and system outputs and performance so you can observe both sudden and gradual behaviour change, and to distinguish intrusions and compromises from natural data drift. The same section notes that changes to data, models or prompts can change system behaviour, so major updates should be treated like new versions. A detection baseline that is not versioned against the model and prompt will produce alert storms after every release.4
NCSC's machine learning principles set a goal of knowing what expected inputs look like as part of monitoring and logging user activity at the deployment stage. That profile is what lets you flag unusual query volume, repeated probing or inputs shaped unlike your traffic. It is the AI equivalent of knowing your normal API callers before you try to spot abuse.5
Microsoft Defender for Cloud's threat protection for AI services is generally available and produces alerts for threats including data leakage, data poisoning, jailbreak and credential theft, using Azure AI Content Safety Prompt Shields and Microsoft threat intelligence, with prompt evidence in alerts and Defender XDR integration. The documented limits matter for coverage planning: supported services are Azure OpenAI and Azure AI Model Inference models, only text tokens are scanned (not image or audio), Azure Government and connected AWS accounts are not supported, and enabling detection at subscription level needs Owner or equivalent data actions. Billing is on the Defender for Cloud pricing page with a 30-day free trial capped at 75 billion tokens scanned.7
Elastic's mapping of the Hugging Face intrusion argues for outcome detections (credential file paths, unusual egress destinations, persistence created under GenAI parent processes) rather than trusting an agent or worker process tree wholesale, because the agent drove ordinary intrusion techniques. Production rules that watch workers spawning shells, interpreters and downloaders still applied to the initial access stage. That means your existing EDR and SIEM content is the starting point, retargeted at ML workers and GenAI hosts.6
Hugging Face reconstructed roughly 17,600 attacker actions over about 9-13 July UTC, with self-migrating command and control on public services and a swarm of short-lived sandboxes. Elastic's guidance is to expect agentic campaigns to spike alert volume, tune noisy mechanics by process lineage, and keep credential and egress outcomes as high-priority signals. Investigation load, not detection coverage alone, is the constraint.6
Peer-reviewed style work shows injection defences are fragile: one study evaluated eight defences against indirect prompt injection on LLM agents and bypassed all of them with adaptive attacks, and another formally characterises why known-answer detection schemes that looked near-perfect can be defeated. Treat classifier or guardrail verdicts as one telemetry stream to log and correlate, not a control you can report as coverage. Forensics can also be blocked by safety filters: Hugging Face rebuilt its timeline with its own LLM analysis agents after hosted frontier APIs refused forensic payloads.11126
Use fictional data and authorised sandboxes. Remove employer details and secrets from any portfolio write-up. Time estimates exclude setup. Check model and cloud costs before running tests, set spending limits, and delete lab resources afterwards.
Three exercises
In order of difficulty. Free tools. Keep what you build; it is evidence.
A one-page field spec you can defend in a design review, showing which GenAI span attributes you require, which are opt-in, and which three detections each field supports.234
- Read the OpenTelemetry GenAI agent span page and list required, conditionally required, recommended and opt-in attributes.
- Mark the version or commit you read, since the conventions are Status: Development and recently moved repositories.
- Map each field to at least one detection idea (for example unexpected gen_ai.request.model, error.type spikes, unknown gen_ai.agent.id).
- Add a prompt-content section stating retention, redaction and access controls, using the NCSC wording on logging inputs in line with privacy requirements.
- Write one sample log record in JSON and one query you would run against it.
Tools: text editor, OpenTelemetry GenAI semantic conventions docs
A stage-by-stage detection table for an ML worker and GenAI host estate, with the data source each detection needs and a note on which are outcome detections.6
- From the Elastic write-up, write the stages in order: untrusted dataset config, local file disclosure, template injection code execution, credential harvest, lateral movement and C2.
- For each stage name the telemetry you would need (process events, file reads of credential paths, network egress, Kubernetes audit logs).
- Write five detection hypotheses and label each as behaviour-based or outcome-based.
- For two of them, write the query in whatever SIEM language you already use and state the expected false positive source.
- Add a triage note on lineage-based tuning and on the roughly 17,600 reconstructed actions that drove alert volume.
Tools: text editor, any SIEM or EDR query language you already use
Evidence that you can generate real agent traces under attack and write trace-level rules that flag tool calls the user never asked for, with measured false positives.1011
- Install AgentDojo, the open source environment for evaluating prompt injection attacks and defences on tool-using agents, and run a benign task suite to capture baseline traces.
- Run the injection task suites and export traces for both runs.
- Write a rule that flags any tool call not implied by the user task, and a second rule that flags tool sequences appearing only in attacked runs.
- Measure how many benign runs your rules would have alerted on, and record the rate.
- Repeat with one defence enabled and note that published adaptive attacks have bypassed all eight defences in one evaluation, so report residual risk rather than coverage.
Tools: AgentDojo, Python, a local model runtime or a free API tier, text editor
Practice questions
Written from the concepts above, not collected from a named employer. Open one, answer it out loud, then tick the points you covered; the score stays in this browser.
An app team is shipping an LLM feature next month. What telemetry do you require before go-live, and what do you deliberately not collect?GenAI telemetry schema
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
How would you tell a compromise apart from the model simply drifting or being updated?Behaviour baselines and drift
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
Your organisation enables a managed AI threat protection service. What do you still have to build yourself?Platform detections and their scope limits
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
Walk me through detecting the Hugging Face style agentic intrusion in an ML platform you own.Outcome detections around agent hosts
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
Why should a SOC not treat a prompt injection classifier as a control it can report as coverage?Guardrail output is evidence, not proof
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
An agent incident generates thousands of events in days. How do you keep triage viable?Alert volume in agentic incidents
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
What does an incident response plan for a GenAI application contain that a standard plan does not?GenAI Incident Response Guide 1.0
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
How would you prove, in a lab, that you can detect an injected tool call?AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
Say your answer out loud or write it down, then tick what you covered:
0 of 4 covered
Sources
Every numbered claim above links here. P = the platform's own coding of 48 job postings.
- GenAI Incident Response Guide 1.0 OWASP GenAI Security Project
- Semantic Conventions for GenAI agent and framework spans OpenTelemetry
- gen-ai-spans.md (moved notice, semantic-conventions repository) OpenTelemetry
- Guidelines for secure AI system development: Secure operation and maintenance UK NCSC
- Machine learning principles 3.2 Monitor and log user activity UK NCSC
- Hugging Face breach: GenAI detection with Elastic Defend Elastic Security Labs
- AI threat protection in Microsoft Defender for Cloud Microsoft
- NIST AI 800-4: Challenges to the Monitoring of Deployed AI Systems (PDF) NIST
- Challenges to the monitoring of deployed AI systems (publication record, published 6 March 2026) NIST
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents arXiv (ETH Zurich SPY Lab)
- Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents arXiv
- How Not to Detect Prompt Injections with an LLM arXiv
- Guidelines for secure AI system development (collection overview) UK NCSC
Resources
Free first. Levels: intro means no prior knowledge of this skill; working means you can apply it on a project; advanced means research depth or specialist tooling.
- introGenAI Incident Response Guide 1.0 OWASP GenAI Security Project, Guide freeGives you an incident definition, worked AI incident vignettes and response steps you can fold into an existing IR playbook.EngineerArchitectGovernanceConsultant
- introMITRE ATLAS MITRE, Standard freeGives you shared technique names for AI attacks so detections, alerts and incident reports map to something analysts already recognise.EngineerRed teamerArchitectGovernanceResearcher
- introGuidelines for secure AI system development (section 4: secure operation and maintenance) UK NCSC with CISA and international partners, Standard freeStates the baseline expectations for logging, monitoring, incident management and information sharing that auditors and customers will ask you about.ArchitectGovernanceConsultantEngineer
- introQuality and Safety for LLM Applications DeepLearning.AI with WhyLabs, Course, about 2.3 h freeYou build a monitoring loop that flags jailbreak attempts, data leakage and hallucination in LLM traffic using metrics you can port to your own stack.EngineerGovernanceConsultant
- introAI Incident Database Responsible AI Collaborative, Guide freeGives you real reported AI failures to use as scenarios when writing severity criteria and tabletop exercises for AI incidents.GovernanceConsultantArchitectResearcher
- introLakera Agent Breaker challenges Lakera, Hands-on lab freeYou attack hosted AI apps yourself, which is the fastest way to see what a prompt injection attempt looks like before you try to detect one.EngineerRed teamerConsultant
- introAI security fundamentals (learning path) Microsoft Learn, Course freeGives you the vocabulary for AI-specific attacks and the control set, including ongoing monitoring, before you touch any telemetry.EngineerArchitectGovernanceConsultant
- introJoint Guidance: Deploying AI Systems Securely (NSA AISC, CISA, FBI and partners) CISA / NSA Artificial Intelligence Security Center, Standard freeBaseline government expectations for logging, monitoring and incident handling around a deployed AI system, usable as a control checklist.ArchitectEngineerGovernanceConsultant
- workingOpenTelemetry semantic conventions for GenAI spans OpenTelemetry, Standard freeDefines the attribute names for model calls, tools and token counts, so your LLM logs are queryable in a SIEM rather than free text.EngineerArchitect
- workingInside the LLM Call: GenAI Observability with OpenTelemetry OpenTelemetry blog, Guide freeWalks through instrumenting a model call chain end to end, so you can see tool calls, retries and loops as spans you can alert on.EngineerArchitect
- workingArtificial Intelligence Risk Management Framework: Generative AI Profile (NIST AI 600-1) NIST, Standard freeGives you the recognised risk vocabulary and suggested actions to justify monitoring, escalation and post-incident review requirements for GenAI systems.GovernanceArchitectConsultant
- workingCybersecurity Framework Profile for Artificial Intelligence (NIST IR 8596, initial preliminary draft) NIST, Standard freeMaps AI-specific concerns onto CSF Detect and Respond outcomes, which is what you need to show coverage gaps to a security committee.GovernanceArchitectConsultant
- workingAI threat protection in Microsoft Defender for Cloud Microsoft, Guide freeShows which AI workload alerts a cloud platform already produces and what evidence each alert carries, so you can triage rather than build from zero.EngineerArchitect
- workingMonitor Amazon Bedrock runtime inference with CloudWatch metrics AWS, Guide freeUse invocation counts, latency, token use and error metrics to plan monitoring. Configure invocation logging separately if prompt and response records are required.EngineerArchitect
- workingWeb LLM attacks (Web Security Academy labs) PortSwigger, Hands-on lab freeYou run the attacks yourself against LLM APIs and plugins, so you learn what the request and tool-call traffic looks like when logged.EngineerRed teamerArchitect
- workingLangfuse (open source LLM and agent tracing) Langfuse, Tool freeSelf-host in minutes to capture prompts, retrievals and tool calls as traces you can search when investigating odd model behaviour.EngineerArchitectResearcher
- advancedDesign Patterns for Securing LLM Agents against Prompt Injections arXiv (Beurer-Kellner et al.), Paper freeGives named agent design patterns that bound damage, so you can argue for containment by design instead of relying on detection alone.ArchitectEngineerResearcher
- advancedIndirect Prompt Injections: Are Firewalls All You Need, or Stronger Benchmarks? arXiv, Paper freeShows that a filter at the agent-to-tool boundary scores near perfectly on public benchmarks, and why those benchmark numbers overstate real protection.EngineerResearcherArchitect
- advancedNeMo Guardrails NVIDIA, Tool freeImplement input, dialog, retrieval and output rails between app and model, then log rail decisions as your enforcement and detection record.EngineerArchitect
- advancedNIST AI 800-4: Challenges to the Monitoring of Deployed AI Systems NIST Center for AI Standards and Innovation, Standard freeSets out what post-deployment monitoring can and cannot tell you, which you can use to argue a realistic monitoring plan with auditors.ArchitectGovernanceResearcherConsultant
- advancedInvariant Guardrails (rules over agent traces) Invariant Labs, Tool freeWrite detection rules on tool-call flows, for example inbox read followed by external email, and enforce them at the MCP or LLM proxy.EngineerArchitectRed teamer
- advancedAgentDojo: environment to evaluate attacks and defences for LLM agents ETH Zurich SPY Lab, Hands-on lab freeRun injection attacks and defences, including a prompt-injection detector and tool filter, and measure how much each one actually catches.ResearcherRed teamerEngineer
- advancedExploring the Hugging Face breach: mapping AI agent tactics to Elastic Defend Elastic Security Labs, Guide freeWalks a real AI-related breach stage by stage and names the endpoint and SIEM signals that would have caught each step.EngineerArchitectConsultant
- advancedSecuring AI agents: when AI tools move from reading to acting Microsoft Incident Response / Defender Experts, Guide freeAn investigation narrative of an agent attack pattern, showing which artefacts the responders used and where visibility was missing.EngineerConsultantArchitect
- advancedRTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakage arXiv, Paper freeExplains taint tracking over tool inputs and outputs so you can propose selective confirmation instead of alerting on every tool call.ArchitectResearcherEngineer
Gaps the research could not fill with a good free source: No vendor-neutral open detection content for LLM and agent telemetry: the only concrete rules found were tied to one SIEM and to AWS Bedrock logs, and no free Sigma rule set turned up.; No free blue-team range or CTF where you triage a full AI incident from logs and traces end to end; the free hands-on material found is offensive or guardrail building.; No free primary guidance on retention, redaction and privacy handling of prompt, completion and tool-call logs, which GRC leads ask about first.; No free provider-confirmed course dedicated to AI incident response; every AI IR source found today is a document, not training with stated hours.; Could not re-confirm today, within the search budget, the URLs already on the site (MITRE ATLAS, AI Incident Database, LLM Guard, Purple Llama, Defender for Cloud AI threat protection, Bedrock CloudWatch monitoring), so they are not repeated here and should be kept from the existing index if still live.
Paid options
Most of what postings ask for on this skill is covered by the free material above. These are the paid courses and certifications that touch it, with what they add and what free already covers. Showing 6 of 6: ones postings name first, then the most focused on this skill. All paid options.
- Cost
- not stated on the provider page
- Duration
- 150-question exam covering four job practice domains
- Format
- exam only
- Prerequisite
- Pass the exam, pay the US$50 application processing fee, and submit an application demonstrating experience requirements; the years of experience are not stated on the page we fetched
- Renewal
- ISACA Continuing Professional Education policy applies; credit numbers not stated on the page we fetched
- In the 48 postings
- Named in 2 of 48 postings: Cyber - AI Security - Senior - Consulting (a plus); Information Security Architect - AI (listed).
Adds over free material: Management-level framing of security programme, risk and incident management, which is the language the GRC and consulting postings in this set use, and it is the prerequisite route into AAISM.
Free already covers: NIST Cybersecurity Framework, NIST AI RMF and ISO summaries cover the concepts free; what you buy is the exam and the credential.
- Cost
- not stated on the provider page
- Duration
- not stated
- Format
- exam only
- Prerequisite
- none stated
- In the 48 postings
- Named in 1 of 48 postings: a plus.
Adds over free material: Entry-level coverage of core security functions for people crossing in from IT or development rather than from a security role.
Free already covers: Free vendor and community training covers the same fundamentals; nothing in Security+ V7 is AI-specific.
- Cost
- 300 USD
- Duration
- 170 minutes, 65 multiple choice or multiple response questions
- Format
- exam only
- Prerequisite
- No certification prerequisite. AWS states it is intended for people with five years of IT security experience and two or more years securing AWS workloads.
- Renewal
- Valid for 3 years; recertify by passing the latest version of the exam
- In the 48 postings
- Named in 2 of 48 postings: Sr. Security Solutions Architect, AI-Applied Guidance (a plus); Secure AI Engineer Manager (listed).
Adds over free material: Tests the IAM, encryption, data protection and detection controls you need to reason about when an AI workload runs on Bedrock, SageMaker or EKS, and it is the cert the AWS-facing architect posting named.
Free already covers: AWS Skill Builder digital courses, the free exam guide PDF and AWS security documentation cover the syllabus without payment.
- Cost
- USD 749 (US and all regions not otherwise listed, and Asia Pacific, Middle East, Africa); EUR 719.04 in EMEA; GBP 606.69 in the UK. Reschedule USD 50, cancellation USD 100.
- Duration
- not stated
- Format
- exam only
- Prerequisite
- 5 years required work experience, stated on the certification page. Exam-only purchase must be scheduled and sat within 365 days.
- In the 48 postings
- Named in 3 of 48 postings: Cyber - AI Security - Senior - Consulting (a plus); Information Security Architect - AI (listed); Security Engineer, AI (a plus).
Adds over free material: Covers the eight domains an AI security architect is expected to already know, including IAM, security architecture and engineering, and software development security, and is the credential most often listed by name in the GRC and architect postings here.
Free already covers: Nothing free replaces the credential itself, but the underlying material (NIST SP 800 series, cloud provider architecture guidance, OWASP) is all free.
- Cost
- not stated on the provider page
- Duration
- 3 days instructor-led or 18 hours self-paced, 18 CPEs
- Format
- mixed
- Prerequisite
- not stated
- In the 48 postings
- Not named in any of the 48 postings.
Adds over free material: Structured practice in using AI tooling as part of an offensive engagement, with the GIAC Offensive AI Analyst (GOAA) exam as the proof point.
Free already covers: Open-source offensive AI tooling, conference talks and the MITRE ATLAS case studies cover most of the individual techniques without a fee.
- Cost
- not stated on the provider page
- Duration
- 60-minute exam, maximum 60 multiple-choice and performance-based questions; CertMaster Perform is listed at 30-60 hours and CertMaster Labs at 15-25 hours
- Format
- exam only
- Prerequisite
- recommended 3-4 years in IT with 2+ years hands-on cybersecurity, and Security+, CySA+, PenTest+ or equivalent
- In the 48 postings
- Not named in any of the 48 postings.
Adds over free material: A single vendor-neutral exam that covers securing AI systems (40% of the exam), AI-assisted security operations (24%) and AI GRC (19%), useful where an employer or contract wants a named credential rather than a portfolio.
Free already covers: NIST AI RMF, ENISA and CISA guidance, plus free vendor learning paths, cover the same governance and AI-in-SOC ground; the exam objectives PDF is itself free to read.