Classification

By the end of this stage the first piece of business logic exists. An LLM-backed classifier looks at the uploaded file and decides whether it is an invoice or a receipt, wrapped as the first step of what will become the processing pipeline.

Why classify first

Document Intelligence has separate prebuilt models for invoices and receipts, but no reliable prebuilt classifier to choose between them. Dave has the agent research it, and the answer comes back a plain no, it does not ship one. The established pattern is classify first, extract second, so the LLM takes the classification job and Document Intelligence keeps the extraction job.

Could one LLM do the whole project? Dave addresses it head on: "yes, but that's not the goal. I want to show you the different types of models and approaches." Using each model for what it is best at is the point of the hybrid design.

Structured output with Pydantic AI

The classifier is a Pydantic AI agent. You declare the output as a Pydantic model and the framework guarantees the response parses into it:

class DocumentKind(StrEnum):
    invoice = "invoice"
    receipt = "receipt"


class DocumentClassification(BaseModel):
    document_kind: DocumentKind
    confidence: float = Field(ge=0.0, le=1.0)
    reasoning: str


class DocumentClassifier:
    def __init__(self, settings: Settings | None = None) -> None:
        resolved_settings = settings or get_settings()
        provider = AzureProvider(
            azure_endpoint=resolved_settings.azure_openai_endpoint,
            api_key=resolved_settings.azure_openai_api_key,
        )
        model = OpenAIResponsesModel(
            model_name=resolved_settings.azure_openai_deployment or DEPLOYMENT_NAME,
            provider=provider,
        )
        self._agent: Agent[None, DocumentClassification] = Agent(
            model=model,
            output_type=DocumentClassification,
            instructions=CLASSIFICATION_INSTRUCTIONS,
        )

    def run(self, document_path: Path) -> DocumentClassification:
        media_type = MEDIA_TYPES.get(document_path.suffix.lower())
        result = self._agent.run_sync(
            user_prompt=[
                CLASSIFICATION_PROMPT,
                BinaryContent(
                    data=document_path.read_bytes(),
                    media_type=media_type,
                ),
            ]
        )
        return result.output

The file itself goes to the model as BinaryContent, the raw PDF or image bytes with a media type. Pydantic AI wraps the text prompt and the binary part into a single multi-part user message, the same thing that happens when you drop a PDF into ChatGPT. A MEDIA_TYPES lookup maps file extensions to MIME types and rejects anything that is not PDF, PNG, or JPEG.

One honest caveat from the video is that LLM confidence scores are not calibrated probabilities. "Unless you very specifically prompt with examples," treat the number as a rough signal, not a measurement. The reasoning field is often the more useful output for a reviewer.

From class to pipeline step

The classifier gets a thin wrapper that fits the step interface the next stage formalizes:

class ClassificationStep:
    name = "classification"

    def run(self, ctx: PipelineContext) -> PipelineContext:
        classification = self._classifier.run(ctx.document_path)
        return ctx.model_copy(update={"classification": classification})

Note the model_copy. Steps never mutate the context they receive; each returns an updated copy. That discipline is what makes the chain easy to reason about once five steps run in sequence.

This step also came out of a refactor. The agent's first version was a scatter of module-level functions, and Dave pushed it into a single class with a clear entry point: "it works, you're using AI, but I'm not really understanding what it's doing." Having clear patterns you steer the agent toward is the recurring lesson of the build.

The branch model

This is also the moment in the video where the repo's branches get their roles. main is the clean starting point you cloned. development is where the build happens, with a checkpoint commit after every stage, "the exact snapshot of where we are." solution is the final result. If you fall behind, you can diff your work against the matching checkpoint, or point your own coding agent at it and ask it to get you to that point.

Test it interactively

cd playground
uv run --project ../backend --locked --no-sync python classify_sample_document.py
# {
#   "document_kind": "invoice",
#   "confidence": 0.98,
#   "reasoning": "The document has an invoice number, VAT IDs, and payment terms."
# }

One gotcha applies to interactive sessions. Pydantic AI's run_sync manages an event loop, and Jupyter-style environments already run one. The fix is two lines at the top of the playground scripts, import nest_asyncio; nest_asyncio.apply(). As Dave says, "this is a little hack. You don't need this in production," which is exactly why it lives in the playground and not in app/.

Checkpoint

• The classifier labels a sample invoice PDF and the fuel receipt PNG correctly
• You can explain why classification uses the LLM while extraction uses Document Intelligence
• You can name the three branches and what each one is for