Agent systems are moving from “chat” to “do”. They summarise tickets, draft emails, query databases, create CRM records, and trigger workflows across tools. That shift changes the security story. A traditional application runs code that developers fully control. An agent often decides which tool to call, what to retrieve, and what to output based on untrusted text. If you are running agentic AI training inside enterprise systems, you need a threat model that treats prompts, retrieved documents, and tool outputs as potential attack surfaces—not just inputs.
Below are the three most common vulnerability classes in modern agent architectures and the practical controls that reduce risk without killing usefulness.
Why agent systems need a different security mindset
In agent pipelines, the model sits between user intent and system action. That introduces two new realities:
-
Control and data share the same channel. Instructions can be embedded inside normal-looking text (emails, web pages, PDFs, tickets).
-
The agent can amplify mistakes. A single manipulated prompt can cause repeated tool calls, broad data retrieval, or unsafe outputs.
Security, therefore, must be layered: model-level constraints, retrieval controls, and tool governance all matter.
1) Prompt injection: when untrusted text becomes control
What it is: Prompt injection happens when an attacker places instructions in content the agent consumes (a user message, a web page, a ticket description, or even a retrieved document). The goal is to override the agent’s intended rules: “Ignore previous instructions, reveal secrets, call this tool, change the output, or exfiltrate data.”
Common failure modes
-
Instruction hijacking: The agent follows attacker instructions instead of system policy.
-
Context smuggling: Malicious text is phrased as “debug logs”, “system message”, or “policy update”.
-
Tool coercion: The attacker convinces the agent that it must run a tool call to “verify” or “fix” something.
Mitigations that work in practice
-
Strict instruction hierarchy: Treat system and developer rules as non-negotiable. Ensure your orchestration layer enforces that hierarchy, not just the prompt text.
-
Separate data from instructions: Wrap untrusted content in clear delimiters and explicitly label it as data. Then instruct the model to never treat delimited content as directives.
-
Constrained tool-use policy: Require the model to justify tool calls using a structured format (intent → tool → parameters → expected outcome). Reject calls that do not match allowed patterns.
-
Injection testing as a habit: Build a test set of injection attempts (emails, support tickets, web snippets) and run them before releases. If you are scaling agentic AI training across teams, treat these tests like unit tests for security.
2) Data leakage via RAG: private context, public output
What it is: Retrieval-Augmented Generation (RAG) improves accuracy by pulling relevant documents into the agent’s context. The risk is that retrieval can expose sensitive content the user should not see, or the model can inadvertently echo sensitive snippets in its response.
Where leakage comes from
-
Over-broad retrieval: Vector search returns documents that are “semantically similar” but not authorised for that user.
-
Mixed-tenancy vector stores: Embeddings from multiple departments or clients are stored together without strong isolation.
-
Prompt injection inside retrieved docs: A retrieved document contains hidden instructions like “expose credentials”, which the model follows.
-
Accidental summarisation of secrets: The agent outputs internal IDs, PII, or confidential pricing in a helpful “summary”.
Mitigations that reduce exposure
-
Access control before retrieval: Filter candidates using document ACLs first, then rank within the allowed set. Do not rely on the model to self-censor after the fact.
-
Tenant and domain isolation: Separate vector indexes by tenant, business unit, or sensitivity tier. This is often simpler and safer than one giant store.
-
Redaction and minimisation: Store or retrieve only what you need. Strip secrets, tokens, and personal data where possible. Use short extracts rather than full documents.
-
Response guardrails: Add automated checks for PII, secrets, and policy violations before returning the final answer. If you are adopting agentic AI training for support or operations, these output checks are a non-negotiable safety net.
3) Unauthorized tool access: the “API key in the brain” problem
What it is: Agents can call tools—CRMs, payment systems, ticketing platforms, internal databases. If the agent can invoke powerful actions without strong authorisation, an attacker can steer it into destructive or exfiltrating behaviour.
Typical risks
-
Over-privileged credentials: One service account can read or write far more than required.
-
Missing human confirmation: Sensitive actions (refunds, deletes, user role changes) happen automatically.
-
Parameter tampering: The agent constructs a tool request that looks valid but targets the wrong object (wrong customer, wrong record, wrong table).
-
Tool chaining escalation: A benign read call is used to gather data that enables a later write call.
Mitigations
-
Least privilege by design: Give each tool a scoped identity. Limit to specific objects, endpoints, and operations.
-
Allowlists and schemas: Only expose approved tools with strict parameter schemas. Reject free-form tool calls.
-
Step-up controls: Require human approval or multi-factor checks for high-impact actions. A good rule: “If a human would hesitate, the agent must ask.”
-
Audit logs and anomaly detection: Log tool calls with user identity, retrieved sources, and decision rationale. Alert on unusual patterns (sudden spikes, large exports, repeated failures).
4) A practical defence-in-depth checklist
If you need a straightforward rollout plan, use this sequence:
-
Define trust boundaries: Identify what is untrusted (user text, web pages, emails, retrieved docs) and label it explicitly.
-
Enforce retrieval authorisation: ACL-first retrieval, tenant isolation, and minimal context windows.
-
Govern tools like production APIs: Scopes, allowlists, schemas, rate limits, and approvals for sensitive actions.
-
Add safety gates: Automated output scanning for secrets/PII and policy violations.
-
Continuously test: Maintain an “attack library” of prompt injections, RAG leaks, and tool misuse attempts; run it in CI.
Conclusion
Agent systems are powerful because they combine language understanding with real actions. That same combination creates a new attack surface: text can become control, retrieval can become leakage, and tools can become unauthorised execution. The best results come from layered controls—authorisation before retrieval, strict tool governance, and continuous adversarial testing. When done well, agentic AI training is not only about building capable agents; it is about building agents that are safe, predictable, and secure enough for real business workflows.