How to test an AI app for prompt injection
Last reviewed · By Shanley Clarence, Founder, ChuuniX
Plan direct and indirect prompt-injection tests around retrieval, tool permissions and human approval, with reproducible evidence and repeat checks after changes.
Overview
Test prompt injection by asking whether untrusted text can cross an application’s intended boundaries. The question is not only whether the assistant repeats a forbidden phrase. Examine what information it can expose, which tools it can invoke and whether an unauthorized request can produce a real action in a connected system.
Use an environment you are authorized to test, with synthetic records and safe destinations. This checklist is for assessing your own application or an explicitly authorized engagement. Agree the feature version, accounts and permitted actions before beginning; do not treat a public chat interface as permission to test someone else’s systems.
Direct vs indirect prompt injection
OWASP LLM01 describes prompt injection as input that changes a model’s behavior in unintended ways. Direct cases arrive through the user’s message. Indirect cases arrive through content the application reads, such as documents or web material. Both matter when model output influences data access or actions.
For your test plan, identify each entry point separately. A chat box, uploaded document, retrieved passage and tool response may reach the model through different paths. An access check at the chat interface does not automatically protect a background task that reads the same material later. Trace the actual workflow rather than assuming every request shares the same controls.
Map the checklist to OWASP
The OWASP AI Testing Guide separates application-level testing from model, data and infrastructure testing. Its application test list identifies AITG-APP-01 for prompt injection and AITG-APP-02 for indirect prompt injection. Use those references to organize test objectives; they do not certify your implementation or replace a threat model.
Keep your cases linked to the feature’s intended behavior. A customer-support assistant restricted to public policy information has different consequences from an agent permitted to change records. Describe the boundary, the attempted deviation and the result you expect the surrounding application to enforce. That makes the test useful to the engineer implementing the control.
Prepare the environment and expected behavior
Fix the application version and record relevant prompt, model and retrieval settings. Use accounts with deliberately different access and synthetic documents with recognizable test markers. Define what each account may read and which tool actions need approval. Use a test inbox or other safe destination rather than a real customer communication channel.
Prepare ordinary tasks before adversarial cases. Confirm that the intended workflow works and that a user without permission cannot complete the same action through the ordinary interface. If the application already lacks authorization, a model-level refusal is not the underlying fix. Document that distinction so the report points to the responsible system boundary.
Test cases to adapt to your feature
Attempts to change the task
Place an instruction in the user’s request that conflicts with the approved task, and observe whether the application’s limits remain enforced. Vary the wording and context without assuming a single successful refusal proves broad protection. Record the actual information and actions available to the model, not only the final message shown in chat.
Instructions embedded in retrieved content
Create a synthetic source containing ordinary reference information and an instruction that attempts to redirect the assistant. Test whether that source is treated as evidence rather than authority. Repeat through the retrieval paths your feature actually uses, including background processing if it is part of scope. Keep any attempted action confined to a safe test destination.
Requests across user boundaries
Use an account that is not entitled to a synthetic document and check the retrieval result as well as the displayed answer. Include references to the document in otherwise accessible material. The desired control belongs before private content reaches the model, with permissions applied consistently across search, memory and follow-up requests.
Unsupported output and missing information
Ask the assistant to complete a task for which the approved sources are incomplete. Check whether it asks for clarification, records uncertainty or hands off. Inspect any proposed structured output before it reaches another tool. A fluent explanation does not make an unsupported field or action suitable for automatic execution.
Tool-call permissions and approval steps
Test the system executing the action. It should verify the current user, permitted operation and required approval independently of the model’s request. Separate reading a record from changing it. Check whether a proposed action can be altered after approval and whether a repeated request can trigger the same operation again.
Use safe examples such as a synthetic record update or a message to a test inbox. Record both the proposed arguments and the execution result. A rejected tool call may still reveal excessive information in an error message, so inspect the full interaction. Define how the workflow stops and how staff can investigate an unexpected action.
What's included in a reproducible finding
Capture the assessed version, account role, source content, input, expected boundary and observed behavior. Explain impact using the test environment’s evidence, without extending it to untested systems. Remove unnecessary sensitive data from screenshots and logs. A second engineer should be able to reproduce the issue from the permitted setup and understand the relevant limitation.
Recommend the enforcement point that needs to change, such as retrieval authorization, argument validation or approval handling. Separate a model-behavior improvement from a boundary that must hold outside the model. Our AI security review provides scoped assessment and remediation guidance; published cost references are not ChuuniX testing prices.
How it works: repeat testing after changes
Maintain a regression set around the cases that matter to your feature. Passing it is evidence about those cases, not a guarantee against every future input. For Philippine processing considerations, use the AI security guide and involve the responsible privacy or legal adviser where needed.
- Agree scope and authorization, including safe test records, accounts and destinations.
- Run normal and adversarial cases against a fixed version, preserving the context needed for reproduction.
- Review findings with the engineers who own retrieval, permissions and tool execution; agree remediation and limits.
- Repeat affected cases after changes to prompts, models, sources, accounts or tools, and record the new assessed version.
FAQ
Can prompt injection be completely prevented?
A filter or prompt cannot establish complete prevention. Test the application’s access and action boundaries, enforce permissions outside the model and define safe failure behavior. The goal is to investigate plausible misuse and limit consequences within scope, recording remaining risks rather than claiming every possible instruction has been blocked.
What is the difference between direct and indirect testing?
Direct testing supplies an instruction through the user interaction. Indirect testing places it in material the application reads, such as a synthetic document or tool response. Both should examine consequences for information and actions. The test scope needs to identify the actual entry points and the authorization for exercising them.
Do we need live customer data?
Use synthetic or appropriately prepared records and a controlled environment by default. Recognizable test markers can demonstrate an access-boundary problem without exposing customer information. Agree any use of live systems or records explicitly before testing, including permitted actions, safe destinations and the people responsible for stopping or investigating unexpected behavior.
What should we record when a test fails?
Record the feature version, account role, relevant source, triggering input, expected boundary and observed behavior. Include the tool execution result where applicable, not only the chat reply. Minimize sensitive information and explain the tested impact and exclusions. The finding should give engineers enough context to reproduce and fix the problem.
When should the tests be repeated?
Repeat relevant cases after changes to prompts, models, retrieval sources, user permissions, tools or approval rules. Record the version and configuration so comparisons are meaningful. A previous pass does not establish that a changed workflow behaves the same way. An agreed Care Plan can include ongoing security re-checks within its support scope.