Measuring whether an AI agent is worth keeping

The question nobody asked at launch

A school group in Vadodara runs five schools and a central admissions office. Between November and March, the office of three people handles thousands of parent enquiries: fees, transport routes, age criteria, documents needed, seat availability, and visit bookings. The group added an admissions assistant on its website and WhatsApp to answer the common questions and book campus visits.

By February, the director asks a simple question: is it worth keeping? The vendor's dashboard shows a large number of conversations handled. The admissions head says her team is still busy. A few parents have complained that the assistant gave the wrong bus route. Nobody can give a straight answer, because nobody wrote down what the office looked like before the assistant arrived.

A few hours of measurement before launch avoids that position and gives a clear decision at the end.

Before launch: take a baseline

Pick a normal two week period and record the following for the process the agent will touch.

• Volume: how many enquiries, invoices, tickets or documents arrive.
• Time: how long each one takes a person, on average, and how long the customer or colleague waits for an answer.
• Quality: how many errors reach the next stage. For admissions, wrong information given to parents. For invoice matching, wrong entries corrected later.
• Backlog: how many items are waiting at the end of each day.

You do not need precise studies. A tally sheet kept by the team for two weeks, plus whatever your helpdesk or Tally already records, is enough. Write down how you measured, so you can measure the same way later.

During the trial: measure four things weekly

1. How much work the agent completes

Count the items the agent handled fully, the items it handed to a person, and the items it got wrong. For the admissions assistant: questions answered without a person, visit bookings made, and conversations passed to the office. Conversation counts alone mean little. A long conversation that ends with the parent calling the office anyway is not a completed item.

2. Quality of what it does

Each week, a person reviews a random sample of the agent's completed work. Twenty to fifty items is usually enough. Mark each as correct, acceptable with small issues, or wrong. Track the wrong ones by cause: missing information in the knowledge source, a misunderstood question, a broken integration. The causes tell you what to fix.

3. Time saved, and where it went

Compare the team's time on the process with the baseline. Then ask where the freed time went. In the admissions office, perhaps the team now calls back every parent who visited, which they never had time for before. That is a real benefit, but it only shows up if you ask.

Be careful with time estimates. If the agent drafts and a person reviews, count the review time. A draft that takes six minutes to check is not a saving over a task that took seven minutes to do.

4. What people think

Ask the team every two weeks: is the agent helping, what annoys you, what should it stop doing. Ask a few customers or colleagues who used it as well. Short, direct answers from the people closest to the work are often the most useful measure you have.

Count the full cost

The cost of an agent is more than its subscription.

• Platform and model usage. Platform fees plus the cost of model calls, which grows with volume and the length of what you send.
• Set up and integration. One time work to connect systems, build the knowledge source and test.
• Ongoing ownership. The hours your own staff spend each week reviewing samples, updating knowledge and handling exceptions. This is often the largest cost and the most often forgotten.
• Cost of errors. A wrong bus route given to a parent costs goodwill and a phone call. Estimate it roughly.

Set these against the time saved and the quality gained. A precise return on investment figure is rarely possible. What you want is a clear answer to whether the work is better, cheaper or faster than before, and by roughly how much.

At ninety days: decide

Put the measures on one page and choose one of three options.

• Keep and extend. The agent does its job well, the team wants it, and the costs are reasonable. Consider giving it a second, related task.
• Keep and fix. The agent helps but has clear problems, such as a gap in the knowledge source or a poor hand over. Fix them and review again in a month.
• Switch off. The agent creates as much work as it saves, or the quality is not acceptable. Turn it off, write down what you learned, and pick a different process.

Switching off is a legitimate result. A strong second project often comes from understanding why the first one did not work.

Limits to plan for

Some benefits are hard to measure: a parent who got an answer at eleven at night and chose your school, a supplier dispute caught a week earlier. Note them, but do not let them carry the whole decision. Equally, the first month is rarely representative. Measure after the agent has had time to settle and the team has learned to work with it.

Checklist

• Baseline of volume, time, quality and backlog before launch
• Weekly counts of completed, handed over and wrong items
• Random sample reviewed each week, with causes of errors
• Review time included in time saved
• Full costs, including your team's ownership hours
• Team and user feedback every two weeks
• A keep, fix or switch off decision at ninety days

To set up a measured trial on one of your own processes, book a demo and bring your baseline numbers.