The hybrid pipeline

By the end of this stage a document goes in one end and normalized, merged, policy-checked data comes out the other, still without any HTTP server or database. This is the central technical lesson of the whole project.

The problem it solves is concrete. Document Intelligence is the stronger extractor but leaves fields empty. The LLM can recover a missing purchase order or amount due, but a general model must not silently replace purpose-built evidence. The answer is a deterministic merge with recorded provenance.

The five steps

The chain is explicit, and each step is a separate piece of code:

1. Azure OpenAI classifies the source as invoice, receipt, or unsupported
2. Document Intelligence runs prebuilt-invoice or prebuilt-receipt accordingly
3. The LLM independently extracts the same requested fields from the source file
4. Pure code compares the two extractions, normalizing names, identifiers, dates, and amounts before judging agreement
5. Document Intelligence wins every conflict; a valid LLM value fills only an empty primary field and is stamped with llm_fallback provenance

Build order

Build it from the inside out, so every piece is testable the moment it exists:

1. The normalized schemas from the playground stage, InvoiceData, InvoiceLineItem, ValidationIssue
2. A small InvoiceExtractor interface, so the pipeline depends on a contract rather than on Azure
3. The Document Intelligence adapter, mapping both prebuilt models into the same normalized shape
4. The LLM document reviewer, returning classification plus a strict-schema extraction
5. The pure merge and reconciliation functions in backend/app/document_review/

Here is the merge in miniature, three fields from the Dutch invoice with an illustrative disagreement:

The first row is a conflict the primary extractor wins. The second is the fallback earning its keep. The third is the important one, the values disagree, the primary value stands, and the disagreement stays visible to the reviewer instead of being resolved silently.

The merge deserves its own sentence. merge_document_extractions is a pure function, and reconcile_invoice_extractions separately records agreements and conflicts. A conflict is shown to the reviewer but never resolves itself by overwriting the primary value. And when a human later corrects a field, the correction updates the normalized data with human provenance while the original provider comparison stays immutable, so edited data can never masquerade as provider evidence.

See it work

Run the representative hybrid evaluation, which sends two known documents through the complete chain:

cd backend
uv run --locked --no-sync python scripts/evaluate_hybrid.py

Watch for two moments in the output. On the Dutch invoice, the LLM fills fields Document Intelligence left empty, and each shows up with llm_fallback provenance. On the fuel receipt, classification routes to the receipt model and the receipt policy. Conflicting non-empty values remain listed as conflicts, with the Document Intelligence value still in place.

Lint the new modules while you are here:

uv run --locked --no-sync ruff check \
  app/invoices/schemas.py \
  app/providers \
  app/document_review

Checkpoint

• A PDF invoice and a PNG receipt both normalize and merge locally, with no FastAPI, SQLite, or React involved
• The hybrid evaluation shows at least one llm_fallback field and routes the receipt correctly
• You can explain what happens when both providers return different non-empty values