Evaluating an AI agent before launch

Why a demo is not enough

A school group in Ahmedabad wants an agent to answer admissions enquiries on WhatsApp during the busy months. In the demo, the agent answers ten questions well: fees for Class 6, the bus route to Bopal, the documents needed for a transfer certificate. Everyone is pleased.

Then someone asks what happens when a parent writes "sibling already studying in grade 3, any concession?" The fee policy has a sibling discount, but only for the second child, only at the same campus, and only for tuition fees. The agent gives a confident, partly wrong answer. Nobody noticed in the demo because nobody asked.

A demo shows that a model can do the task. An evaluation shows how often it does the task correctly on the work you actually get, and where it fails. You need the second before launch.

Build the test set from real past work

The best test set is last month's real cases, with what your team actually did on each one. For the school, that is the enquiries that came in during September and the answers the admissions office sent. For an invoice agent, it is last month's supplier invoices and the entries your accountant posted.

Aim for thirty to fifty cases for a first evaluation, and more for high volume work. Pick them so they reflect reality:

• Mostly routine cases, in roughly the proportion you see them. If seventy per cent of enquiries are about fees, about seventy per cent of the test set should be too.
• The awkward ones on purpose. The sibling discount, the transfer mid year, the invoice with a rate change, the scan taken at an angle. Ask the person who does the work which cases they dread.
• Cases the agent should refuse or hand over. A parent's complaint about a teacher. An invoice from a vendor who is not in the master. A good agent knows when to stop.

Remove or mask personal details you do not need for the test, and keep the set somewhere with the same access rules as the live data.

Decide what correct means before you run it

Write down, for each case, what a correct result looks like. This is harder than it sounds, and it is where most of the value lies.

For extraction tasks it is usually exact: the GSTIN, the invoice number, the total. For answers it is about content: did it give the right fee, cite the right policy, avoid promising anything it should not? For routing it is the destination: did the ticket go to the right queue?

Agree a simple scale with the process owner. Three grades are enough for most work:

• Correct: a person would approve it as it is.
• Fixable: the gist is right but a person would need to edit it.
• Wrong: a person would reject it, or it would cause harm if sent.

Record a fourth outcome separately: handed over. When the agent says "I am not sure, passing this to the office", that is often the right behaviour, so score it on its own line.

Run it and read the failures

Run the agent on every case in a test workspace, with the same instructions, sources and tools it will have in production. Then compare its result with what your team did.

The overall score matters less than the pattern of failures. Read every wrong and fixable case and ask why:

• Missing source. The answer was not in the documents the agent could see. The fix belongs in the knowledge source.
• Wrong source. It found an old fee schedule. Remove or label outdated documents.
• Misread input. The scan was poor or the layout unusual. Add a confidence check that sends such cases to a person.
• Rule not written down. The team applies a rule that exists only in someone's head. Write it down; that alone often improves the process.
• Model limitation. Rare, but real, for example with certain scripts or very long documents. Try a different model or split the task into smaller steps.

Fix, rerun the whole set, and compare. Keep every run so you can see whether a change helped one case and broke two others.

Agree the launch decision in advance

Before the first run, agree with the process owner what result is good enough to start, and with what approvals switched on. For example:

• Wrong answers on routine cases must be rare, and every wrong answer must be one a reviewer would catch.
• Every case that involves money, a complaint or a policy exception must reach a person.
• When a source is missing, the agent must hand over to a person.

There is no universal threshold. An agent that drafts replies for a person to send can launch earlier than one that sends on its own. The point is to decide the bar before seeing the score, so the score cannot move the bar.

Keep evaluating after launch

Plan to rerun the test set. Models are updated, policies change and new kinds of cases arrive. Run the set again:

• before any change to instructions, sources or model goes live,
• when the provider releases a new model version you plan to use,
• every month or quarter, with new real cases added from what reviewers corrected.

In AksharAI, test runs live in the agent studio next to the agent itself, so the same people who change the rules can see the effect. See the agent studio and where a person must approve for how the pieces fit together.

A short checklist

• Thirty to fifty real past cases, including the awkward ones and the ones to refuse
• A written definition of correct for each case
• Three grades plus handed over, agreed with the process owner
• Every failure read and given a cause
• A launch bar agreed before the first run
• A plan to rerun the set on every change

If you want a quick sense of whether a process is ready for this kind of testing, the agent readiness check asks ten questions and suggests what to prepare first.