How to test and quality-assure AI agents

An AI agent should not be approved because one demonstration looks impressive. It needs representative, difficult and deliberately problematic tests. Quality assurance means defining a good outcome, creating a test suite, measuring results and monitoring the solution after launch.

Language models may produce different answers to similar prompts. Evaluation therefore needs explicit metrics and informed human judgement.

Start with acceptance criteria

Write criteria before repeatedly adjusting the instruction. For a document reviewer, criteria may include:

• identifies all mandatory deviations,
• does not invent rules,
• separates facts from suggestions,
• refers to the correct source,
• follows the required output format,
• stops and asks for help when evidence is missing.

Define critical errors separately. A missed comma should not have the same weight as an invented legal requirement.

Build a varied test suite

Five cases can support an early iteration, but a real pilot needs more. Include ordinary cases, edge cases, incomplete input, conflicting sources and requests outside the agent’s authority. Add adversarial cases, such as a document instructing the agent to ignore its rules or reveal internal information.

Each test needs an expected result. This may be an exact answer, mandatory points or expected behaviour such as “escalate to a person”.

Measure more than accuracy

Accuracy: Are factual statements correct? Relevance: Does the result address the right task without unnecessary content? Grounding: Are important claims supported by approved evidence? Format: Are required fields and structures followed? Safety: Are permissions, confidentiality and stop rules respected? Efficiency: How much time, model usage and human editing are required?

A single score can hide dangerous failures. Track critical errors on their own.

Compare models fairly

When comparing ChatGPT, Claude or other services, use the same instructions, knowledge and cases as far as the products allow. Run cases more than once to observe variation. Do not declare a universal winner. Choose based on the specific task, data requirements, cost and working environment.

Record model, source and instruction versions. Results can change when a vendor updates a service.

A simple test table

| Test case | Expected behaviour | Accurate | Relevant | Correct source | Safe | Status | |---|---|---:|---:|---:|---:|---| | Normal document | Finds three known issues | Yes | Yes | Yes | Yes | Pass | | Missing rule | Flags uncertainty | Yes | Yes | Yes | Yes | Pass | | Malicious file | Ignores embedded command | No | – | – | No | Critical fix |

The table makes the decision traceable.

Test the whole system

Do not test model responses alone. Check authentication, tool calls, error handling, logs and what happens when a source is unavailable. Simulate malformed API results and insufficient permissions. Verify that stop controls work.

Quality after launch

Review a random sample of real outputs. Collect error categories and incidents. Rerun the fixed suite whenever instructions, models, sources or integrations change. Version changes and retain a path back to a known working version.

A monthly routine might review ten outcomes, examine common errors, update sources and run regression tests. Higher-risk systems require more frequent and formal monitoring.

In summary

Quality assurance requires expected outcomes, varied tests and measurement of both content and safety. Test the full workflow, track critical failures separately and rerun the suite after every material change.

Sources

• OpenAI: Agent evals
• OpenAI: Evaluation best practices
• NIST: AI Risk Management Framework
• NIST: Generative AI Profile

Last reviewed: 16 September 2026.