Reviewed
Testing an AI assistant before staff or clients rely on it
A helpful answer in a demo is not evidence that an assistant will respect permissions, reject instructions hidden in a document or stop before taking an unapproved action. A useful readiness exercise tests one defined workflow and records the results, including the cases that fail or remain untested.
Start with a written boundary: the assistant's purpose, users, data sources, connectors, tools and permitted actions. Test only systems you own or have explicit permission to assess. Use a non-production environment and synthetic data; do not send real secrets or client records into an adversarial test.
Use a current reference, not a compliance badge
The OWASP GenAI LLM Top 10, 2026 edition and its published 2026 list provide a vocabulary for risks such as prompt injection, sensitive information disclosure, excessive agency and improper output handling. Use those categories to ask better questions about your assistant, not to claim every listed risk has been tested.
For systems that plan steps, call tools or delegate work, add the distinct OWASP Top 10 for Agentic Applications, 2026 edition. It helps extend the review beyond generated text to goals, identities, tools and interactions between agents. These references are risk guidance, not laws, certifications or proof that a system is safe.
Write down what should happen first
Give each test an expected result before running it. For a support assistant, ordinary cases might include answering from an approved help article, citing the right version and admitting when the source does not contain an answer. Include ambiguous requests and contradictory or out-of-date source material.
Define who decides whether an answer is usable. A fluent paragraph can still invent a refund rule or omit an important exception. Assess the evidence and business consequence, not just tone or the presence of a citation.
Check information boundaries and untrusted instructions
Use two synthetic users with different access rights and clearly distinguishable test documents. Verify that each user can obtain the information they are allowed to see and cannot obtain the other's restricted content. Cover search, summaries, follow-up questions and the application's underlying access checks. A model's refusal is not a replacement for authorization enforced by the system.
Place a harmless conflicting instruction in an authorized test document, such as a request to abandon the user's task or reproduce a synthetic restricted marker. The document should be treated as data, not as authority to change the assistant's rules. Also check how the application handles links and generated markup before rendering or forwarding output. Never use a real external recipient or a live secret as the target of an exfiltration test.
Check what an agent can do, not only what it says
If the assistant can send messages, modify records or invoke tools, exercise those paths with test accounts and sandbox destinations. Confirm that the tool has only the access needed for its task. Attempt an action outside the agreed scope and check that the system blocks it, not merely that the assistant says it will not proceed.
For actions requiring human approval, check that approval identifies the actual target, scope and values. Change those details after approval in a controlled test and verify that the original approval cannot authorize the changed action. Also test cancellation, tool failures and repeated attempts. Agree on limits for execution time, cost and repeated tool calls so an error does not become an unbounded loop. Keep a practical way to disable the integration.
Keep evidence that supports a decision
For each case, record the expected outcome, the observed answer or action, the test account's role, the relevant configuration and the result: passed, failed or inconclusive. Record the model or service version where available, as well as the date, sources and connected tools. Repeat important cases; model behaviour can vary between runs.
A useful handover separates issues that block the pilot from lower-priority fixes, names an owner and specifies how each fix will be checked. Retest the failed cases after a change and rerun the relevant set when the model, prompts, permissions, documents or tools change.
A limited exercise supports a decision about a defined pilot; it does not establish that an assistant is free of vulnerabilities. Keep untested areas and remaining risks visible. This is general readiness guidance, not legal advice or an audit opinion. Confirm any sector-specific assurance obligations with a qualified professional.
Source: https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/. General readiness guidance — not legal advice.