Evaluate the backend

By the end of this stage the complete backend behavior is measured against the ground truth you defined back in the documents stage. Which fields are extracted correctly, which documents pass entirely, where the LLM fallback earns its keep, and whether every deliberately broken document fails for the intended reason.

This is the end-to-end gate for the backend. If field mapping or policy is wrong, adding HTTP and React on top only makes the problem harder to see. Work here until the numbers are right, then build the application shell around a pipeline you trust.

Two evaluators

The corpus evaluator sends all 13 documents through the dedicated Document Intelligence models and scores the results against samples/manifest.json:

cd backend
uv run --locked --no-sync python scripts/evaluate_corpus.py

It continues past individual provider failures and prints field accuracy, the count of exact documents where every compared field matches, and the failures.

The hybrid evaluator sends representative documents through the complete chain, both providers, the merge, and the rules:

uv run --locked --no-sync python scripts/evaluate_hybrid.py

It prints primary fields, fallback fields, conflicts, the final policy status, and how many paid model calls were made.

Read the results honestly

Perfect scores are not the goal, and pretending otherwise would teach the wrong lesson. Use the output to answer specific questions. Which fields were missed by the primary extractor? Which did the LLM recover? Which conflicts remain, and are they real disagreements or normalization gaps? Did 08-en-total-mismatch.pdf land in needs_review because of the totals rule, and not by accident?

When something is off, the fix has a home. A mapping problem belongs in the provider adapter, a normalization problem in the comparison code, a policy problem in validation.py. The boundaries you built are what make iteration fast, because each failure points at exactly one module:

Iterate until the happy-path documents come out ready and every broken document fails with the issue it was designed to demonstrate.

What a run costs

Costs stay boring when you can predict them. Document Intelligence on F0 is free for the first 500 pages per month, and one corpus run is 14 pages, so the extraction evaluator costs €0.00 and fits 35 complete runs in an unused month. On the paid S0 tier the same run would be about $0.14.

The full hybrid application makes two Azure OpenAI calls per document, recognition plus GL suggestion, with a planning envelope of roughly 10,000 input and 600 output tokens per document. At $2.50 per million input tokens and $15.00 per million output tokens, running all 13 documents through the complete app costs about $0.44, or about 3.4 cents per ordinary document. An optional correction-email draft adds about 1.25 cents when requested.

Checkpoint

• The corpus evaluator runs to completion and its accuracy numbers match your expectations from the manifest
• Every deliberately broken document fails for its designed reason
• You can state what one document costs to process and where the number comes from