Closing the loop

By the end of this stage the app delivers Maya's full user story. She can correct extracted fields, approve or reject documents, and draft a correction email to the supplier. Under the hood, the hybrid extraction promised on the architecture page finally runs, two independent readings of every document, merged with provenance.

The prompt that closes the loop

This stage is one deliberate big jump. The prompt asks the agent to close the loop on the application layer back to the user story, to let Maya edit fields, approve, and reject, to consult the sample manifest for the errors worth surfacing, and to make sure everything passes the evaluation dataset. The reason Dave is comfortable jumping this far is the manifest itself, and that is the teaching heart of the stage.

Labeled data as the goal

samples/manifest.json records, for every corpus document, the expected field values and the expected issue codes. Documents 1 through 4 should come out green. Document 5 should raise vendor_vat_id_required. The mismatch invoice should raise invoice_total_mismatch. That turns a vague instruction like "make review work" into a checkable target: "you can essentially give your AI agents a goal to start working and optimizing towards." The scripts backend/scripts/evaluate_corpus.py and evaluate_hybrid.py score the pipeline against the manifest.

The advice generalizes beyond this build. Dave tells a war story about a client document pipeline that started from three happy-path examples and drowned in errors once real documents arrived: "you never one-shot an application of that scale." Whenever you start a document project, collect and label data with different scenarios first. The corpus with its deliberate failure cases was designed before any code, with this moment in mind.

The hybrid review step

The new pipeline step slots between extraction and validation, in the gap the chain design left:

class DocumentReviewStep:
    """Project DI extraction, run independent LLM review, and merge gaps."""

    name = "document_review"

    def run(self, ctx: PipelineContext) -> PipelineContext:
        review_data = project_extraction(ctx.extraction)

        llm_extraction = self._reviewer.review(ctx.document_path, content_type)
        if llm_extraction.document_type == "unsupported":
            raise UnsupportedDocumentError(
                "The uploaded file is not a supported invoice or receipt."
            )

        merged, document_review = merge_document_extractions(review_data, llm_extraction)
        return ctx.model_copy(
            update={"review_data": merged, "document_review": document_review}
        )

Three moves. First, project_extraction in backend/app/documents/projection.py flattens the nested Document Intelligence extraction into ReviewData, a flat structure of the fields Maya reviews, carrying a field_confidence and a field_sources map per field. Second, an independent reviewer in backend/app/providers/azure_openai_document_review.py sends the same file to Azure OpenAI with a strict JSON schema, a second reading that never saw the first. Third, the merge.

The merge rule is the one from the client brief, and it is worth stating precisely. Document Intelligence stays primary. An LLM value can fill a field Document Intelligence missed, but it can never replace a value Document Intelligence found. Filled gaps are tagged llm_fallback in field_sources, so the UI can show the reviewer which values came from where.

Reconciliation in backend/app/document_review/reconciliation.py also compares the two readings field by field. Each of the thirteen tracked fields gets a normalizer suited to its type, text, identifier, date, or amount, so EUR 847,00 and 847.00 compare as equal, and a FieldComparison records whether the readings match, differ, or are missing on either side. Agreement between two independent extractors is cheap evidence the value is right; disagreement is exactly what a reviewer should look at.

The review UI

On the frontend, DocumentReview.tsx and DocumentReviewSection.tsx render the projection with its issues and sources. Every field is editable until a decision is made. An edit calls PUT /api/documents/{id}, the field's source becomes human, and policy re-runs, which also means the low-confidence warning for that field disappears, human input outranks model confidence. Approval is gated in backend/app/documents/service.py. A document with open errors cannot be approved, and a decided document can never be edited again.

Walking the corpus through the UI in the video, a clean invoice shows a single confidence warning, approved, into history. Document 5 shows the missing VAT error, and the approve button stays locked. "I cannot approve this. That's exactly how we want it."

The correction email

Rejection has a cherry on top. When issues are the supplier's to fix, the UI offers to draft a correction email. Eligibility is code, not model judgment, a fixed set of issue codes in backend/app/correction_email/eligibility.py:

SUPPLIER_FIXABLE_CODES = {
    "vendor_vat_id_required",
    "vendor_vat_id_invalid",
    "invoice_total_mismatch",
    "purchase_order_missing",
    # ...every code a supplier can actually act on
}

Only when supplier-fixable issues exist does POST /api/documents/{id}/correction-email hand the document context to the LLM to draft a polite request, rendered in CorrectionEmailDialog.tsx with a copy button. The app deliberately never sends mail, though as Dave notes, an integration that does is the obvious extension.

The receipt path

The last requirement check is the fuel receipt PNG end to end, classified as a receipt, extracted with prebuilt-receipt, validated against the shorter receipt rule set, coded to travel and transport, all green, approved. Every line of the functional requirements list now has its checkmark, and Dave calls it. The project is deliverable. With one caveat he insists on, the first delivered version is never the last: "this is never going to be the version where users are totally okay with." Feedback that tweaks review flow is in scope; Maya asking for Word documents is a new functional requirement to scope separately.

Checkpoint

• A failure document shows its issues, and approval stays locked until they are resolved
• Editing a field marks it human, clears its confidence warning, and revalidates
• The correction email drafts only for supplier-fixable issues, and the receipt processes end to end
• evaluate_corpus.py scores the pipeline against the manifest expectations