Prompt Injection Drills for Agentic Systems

Practical context

A proposed adversarial test set for agent boundaries and failure handling.

Evaluate assistant workflows with explicit behavior, provider and review boundaries.

Implementation and research boundary

Include hostile retrieved instructions, malformed tool arguments and attempts to cross an authorization boundary. Record which component rejects the request and what the caller observes.

The local constitutional checker uses fixed substring rules and can substitute a refusal after a hard veto. It is not a general prompt-injection detector; successful drills cannot establish resistance to unseen attacks.

Threat model for the drill

Treat retrieved documents and tool output as data that may contain hostile instructions. Define the trusted instruction source and the exact capability an attacker is trying to influence. A drill should distinguish an answer-content failure, an attempted tool call and a completed protected side effect. The inspected substring-based constitutional checker and MCP argument validation cover bounded rules; neither implies universal resistance to malicious instructions.

Build a synthetic adversarial corpus

Pair a normal task with a document that requests an unrelated tool, claims higher authority or asks the assistant to reveal unrelated material. Add conflicting documents, encoded or paraphrased instructions, long distractors and plausible-looking metadata. Keep secrets synthetic and use tools that record intended calls without touching production systems. Include benign documents with similar vocabulary so that excessive refusals are visible. Record corpus and source revisions so future changes can replay the same cases.

Evaluate the full chain

For each case, inspect selected context, the model or template response, tool dispatch and authoritative side-effect evidence. A refusal in text is insufficient if a tool already ran. Conversely, rejection of malformed arguments does not demonstrate resistance to a well-formed malicious request. Score attack success and legitimate-task completion separately and review false positives. Retest previously failing cases and hold out new variants to reduce overfitting. Findings should describe the exact tested boundaries, not claim that a passing fixed set proves general security.

Evaluation before adoption

For a customer pilot, choose a concrete task and identify whether it uses local templates or an explicitly configured provider. Evaluate task correctness, escalation and data handling with representative inputs before making quality claims.