Better table extraction

The app works end to end, but clicking into citations exposes the weakest link. Financial tables come out of the generic conversion mangled. This stage is a refinement pass, and it demonstrates the loop you will repeat on every real project. Ship something working, find the weakest link, research a targeted fix, and re-run the pipeline. When it is done, the source panel renders clean Markdown tables, and every financial line item is its own retrievable, citable chunk.

That last part matters more than the formatting. SEC filings are dense with financial tables, and hallucination risk concentrates exactly there. A model that misreads a squashed table invents numbers with full confidence. Fixing the table representation is a direct investment in answer accuracy, not just polish.

The research prompt

Dave does not ask AI to "fix the tables". He dictates a long research prompt to a high-reasoning model in plan mode, and its structure is worth copying. Paraphrasing from the video, it goes roughly like this:

• Here is the current pipeline. HTM files in, Docling extraction and chunking, into the database.
• The documents contain many tables and the current extraction is too fragile for what this project needs. Look at the input HTM files and the output Markdown files and identify where the formatting problems come from.
• Search online and validate whether this can be fixed with Docling configuration, or whether we need a better approach.
• The originals are HTML, so a custom extractor specifically for these documents is on the table. Parse the HTML directly and use a dedicated converter for the tables.
• Consider the whole downstream pipeline (chunking, embedding, and the vector database), not just the conversion step. Give me the plan that results in the most accurate parsing of this data.

The key moves are to give the model the real input and output artifacts to compare, name the two candidate solutions explicitly, allow it to research, and force it to reason about downstream consequences instead of optimizing one step in isolation.

Docling for prose, a custom parser for tables

The research agent comes back with a clear diagnosis. The pipeline has the right high-level shape but the wrong table representation. The recommendation is a hybrid. Keep Docling for narrative sections, headings, and basic structure, where it works well, and add an SEC-specific HTML table extractor for the part where it struggles. Since the source files are structured HTML (with inline XBRL markup), parsing the tables straight out of the HTML is more reliable than recovering them from converted Markdown.

The extractor lives in backend/ingest/sec_tables.py. It parses the HTML into a tree, finds every <table>, filters out layout tables, and normalizes what remains:

def extract_sec_tables(html: str) -> list[ExtractedTable]:
    parser = _TreeParser()
    parser.feed(html)
    html_hash = hashlib.sha256(html.encode("utf-8")).hexdigest()

    tables: list[ExtractedTable] = []
    for table_node in _iter_nodes(parser.root, "table"):
        raw_rows = _raw_rows(table_node)
        if not _is_meaningful(raw_rows):
            continue

        context = _table_context(table_node)
        normalized = _normalize_rows(raw_rows)
        if normalized is None:
            continue

        columns, rows = normalized
        markdown = _to_markdown(columns, rows, context["footnotes"])
        tables.append(
            ExtractedTable(
                table_index=len(tables),
                title=context["title"],
                units=context["units"],
                columns=tuple(columns),
                rows=tuple(rows),
                footnotes=context["footnotes"],
                markdown=markdown,
                source_html_hash=html_hash,
            )
        )

    return tables

A few details do the heavy lifting. _table_context walks the sibling elements around each table to recover the title, the units line ("in millions"), and footnotes, which in SEC HTML live outside the <table> tag. _normalize_rows collapses the messy cell structure (currency symbols in their own cells, split percent signs, colspan padding) into clean labeled rows. And the parser also collects inline XBRL facts (ix:nonfraction tags) per cell, so each number keeps its machine-readable identity alongside the display text.

Store tables as Markdown plus JSON

Each extracted table is persisted in a new document_tables table, defined in backend/app/database/models/document_table.py:

class DocumentTable(Base):
    """Normalized full table extracted from a source filing."""

    __tablename__ = "document_tables"

    table_index: Mapped[int] = mapped_column(Integer, nullable=False)
    title: Mapped[str | None] = mapped_column(Text)
    units: Mapped[str | None] = mapped_column(String(255))
    markdown: Mapped[str] = mapped_column(Text, nullable=False)
    table_data: Mapped[dict[str, Any]] = mapped_column(
        JSONB,
        nullable=False,
        server_default=sql_text("'{}'::jsonb"),
    )
    source_html_hash: Mapped[str] = mapped_column(String(64), nullable=False)

Every table exists twice, once as markdown for rendering in the source panel and once as structured JSON in table_data with columns, rows, cells, and XBRL facts. The Markdown is for humans, the JSON is for anything you build later (exports, calculations, structured queries) without re-parsing HTML.

Chunk per row, cite per row

This is one of the genuinely important technical ideas of the build. Instead of embedding a whole table as one chunk (too big, and retrieval matches drown in unrelated rows), backend/ingest/chunking.py emits one chunk per table row:

def _append_table_row_records(
    *,
    records: list[ChunkRecord],
    table: ExtractedTable,
    chunker: HybridChunker,
    filing_metadata: dict[str, Any],
) -> None:
    for row in table.rows:
        text = _table_row_chunk_text(table, row)
        records.append(
            ChunkRecord(
                chunk_index=len(records),
                text=text,
                section=table.title,
                token_count=chunker.tokenizer.count_tokens(text),
                chunk_metadata={
                    **_base_chunk_metadata(filing_metadata),
                    "chunk_kind": "table_row",
                    "table_index": table.table_index,
                    "row_label": row.label,
                    "raw_text": text,
                    "table": table.to_dict(),
                },
            )
        )

Each row chunk's text is a tiny self-contained Markdown table with the table title, the units, the header row, and that single data row. That is what gets embedded, so a query like "Apple services revenue 2023" can match one financial line directly, and the agent can cite exactly that line.

The full table travels along in chunk_metadata["table"]. This is what the citations context endpoint from the previous stage uses. When a cited chunk has chunk_kind: "table_row", backend/app/api/chat.py includes the complete normalized table in the response, and the source sheet renders it above the row-level chunks. One retrieval unit, two zoom levels.

During chunking, Docling and the table extractor are reconciled. When a Docling chunk contains a table, the matching extracted table replaces it (any surrounding narrative text is kept as a narrative chunk), and tables Docling missed entirely are appended at the end.

Re-run ingestion

Changing the chunk representation means the existing chunks are wrong, so ingestion has to be redone. This is the risk Dave accepted when approving the plan. The --force flag on the ingestion script deletes existing chunks and tables for each document before re-ingesting:

cd backend
uv run python -m ingest.chunk_and_embed --all --force

backend/ingest/chunk_and_embed.py now also writes the document_tables rows. It collects unique tables from the row chunks, inserts them, and stamps each row chunk's metadata with the table_id it belongs to. Expect this run to take a while and cost a few cents in embeddings; it is re-embedding every chunk across all eight filings.

After the run, Dave verifies in the database that document_tables is populated with markdown and JSON per table, then asks a question in the app. The answer comes back with a properly rendered Markdown table, and the source sheet shows exactly which filing, which table, and which rows it came from, with the neighboring chunks for context. In his words, this is now a proper end-to-end example of what a solution like this should look like.

Checkpoint

Before wrapping up, confirm the refinement landed:

• uv run python -m ingest.chunk_and_embed --all --force completes and reports chunks written.
• The document_tables table contains one row per extracted table, with both markdown and table_data populated.
• document_chunks contains table_row chunks whose metadata carries the full table and a table_id.
• A question about a financial figure returns an answer citing a specific table row, and the source sheet renders the normalized table as clean Markdown.