Document extraction accuracy: what to measure and how

One number hides the story

A hospital in Surat wants to extract fields from discharge summaries for insurance claims: patient name, admission and discharge dates, diagnosis, procedures, and the final bill amount. A vendor says their system is highly accurate. That claim means little until three points are clear:

• Whether accuracy is counted per document or per field.
• Whether it was measured on typed summaries or on scans with handwritten notes.
• Whether it covers the bill total, which matters most, or mostly easy fields such as the hospital name.

An extraction system can be right on ninety five fields out of a hundred and still be wrong on the one that decides the claim. That is why you measure on your own documents, field by field, and design the process for the errors.

Measure per field, on your own documents

Build a test set of real documents, as described in evaluating an AI agent before launch. For extraction, fifty to a hundred documents across your main sources and layouts is a reasonable start. For each document, record the correct value of every field you need.

Then measure each field separately:

• Exact match rate: how often the extracted value is exactly right. Use this for identifiers such as GSTIN, PAN, invoice numbers and policy numbers, where close is wrong.
• Normalised match rate: right after normalising format, for example dates written as 12/10/26 and 12 Oct 2026. Use this for dates and names.
• Tolerance match: within an agreed tolerance, for amounts where rounding differs.
• Missing rate: how often the field was not found at all.

A table of these per field tells you far more than one overall figure. You will usually see that a few fields cause most of the trouble.

Weight the fields by what they cost

Not every error matters equally. A wrong hospital address on a discharge summary is a nuisance. A wrong bill amount or diagnosis code can delay or reject a claim. Mark each field as critical, important or minor, and set a target for each group. Critical fields should either be right or be flagged for a person; they should never be silently wrong.

Measure the safety net as well as the extraction

The most useful measure is often this one: of the values that were wrong, how many were caught before they caused harm?

Errors get caught in three ways:

1. Confidence checks. The system marks values it is unsure of, and those go to a person.
2. Rule checks. Values are checked against each other and against other systems. The invoice total should equal the sum of the lines plus tax. The GSTIN should match the vendor master. The discharge date should not be before the admission date.
3. Human review. A person approves the record before it is used.

A process where extraction is right ninety per cent of the time and almost every error is caught by checks is safer than one that is right ninety eight per cent of the time with no checks. Measure the share of errors caught, and push it as close to all of them as you can for critical fields.

Know your document mix

Accuracy varies a lot with the source:

• Digital PDFs generated by software are the easiest.
• Clean scans are close behind.
• Phone photos vary with light, angle and folds.
• Handwriting varies with the hand; neat block capitals are often fine, hurried notes often are not.
• Unusual layouts from small suppliers or old forms cause more misses.

Measure each source separately. If photos from one supplier fail often, the fix may be asking that supplier for a PDF.

Design for the failures

Assume some documents will fail and decide in advance what happens:

• Low confidence on a critical field: send to a named reviewer with the image region highlighted.
• Rule check fails: show the reviewer which rule and the values involved.
• Document unreadable: ask the sender for a better copy, automatically if the channel allows.

The document extraction agent on AksharAI works this way: extract, check, and route anything uncertain to a person before it enters your system.

Keep measuring after launch

Every correction a reviewer makes is a new test case. Add a sample to the test set each month and rerun it when the model or the instructions change. Watch for new layouts, such as a supplier changing their invoice software, which can quietly lower accuracy on one source.

For invoice extraction, AI for Tally users shows how it fits into an accounts team's day.