Document ingestion

By the end of this stage the 25 SEC filings are in Supabase twice, once as full Markdown documents in source_documents and once as embedded chunks in document_chunks with page numbers, section headings, token counts, and 1536-dimension vectors. This is the corpus every later retrieval feature searches against.

The raw downloads are .htm files full of XML noise, so the pipeline converts HTM to Markdown with Docling, loads the results into the database, chunks with structure awareness, and embeds with OpenAI. Each step is a small script you run once, locally.

Keep ingestion out of the application

Docling is the conversion workhorse here, an open source library that turns messy documents into structured data, detecting tables and layout along the way. It runs locally on ML models, which makes it heavy, and the app in production never needs it. So it is added as an optional extra, not a main dependency:

# backend/pyproject.toml
[project.optional-dependencies]
ingest = [
    "docling==2.96.0",
]

cd backend
uv sync --extra ingest

The same reasoning shapes the folder layout. The conversion script lives in data/, next to the downloads, and the chunking scripts live in backend/ingest/, outside backend/app/. Dave's rule is "anything that's not required for your deployment, keep it out of your application folder, because it's code your app doesn't rely on." One-off local scripts should not ship with the API.

Convert HTM to Markdown

data/convert_to_markdown.py walks the download manifest and converts each filing, mirroring the original folder structure (per ticker, per year) into data/markdown/. Strip away the path management and the core is two lines:

from docling.document_converter import DocumentConverter

converter = DocumentConverter()

for filing in filings:
    result = converter.convert(html_path)
    markdown = result.document.export_to_markdown()
    md_path.write_text(markdown, encoding="utf-8")

The script also writes a new manifest.json into data/markdown/ that carries the filing metadata (ticker, CIK, form, dates, accession number, source URL) forward, plus a SKIP_EXISTING flag so re-runs do not reconvert finished files.

uv run data/convert_to_markdown.py

Inspect what came out

Open a converted file and scroll through it. The narrative text converts cleanly almost everywhere; the tables are where you look hardest, because a single column shifted one position sideways corrupts every number in it. In this corpus the tables are mostly fine, with some duplicate columns and odd formatting here and there.

Dave's warning deserves its own paragraph, "in the real world, it could very well be that this whole ingestion process is the step where you spend the most time." Everything downstream inherits whatever errors happen here. On a paid project he would spend a day or two experimenting with different extraction methods before moving on.

To get a second pair of eyes without stopping the build, he spawns a parallel reviewer agent with a reasoning-heavy model and prompts it, "you are a document reviewer, spot-check some of the tables and tell me if Docling converted them correctly." It comes back later. Factually correct, some duplicate columns, faithful enough for a RAG pipeline. Good enough for now, noted for later.

Load filings into source_documents

backend/ingest/load_source_documents.py reads the Markdown manifest and writes one source_documents row per filing. Since this is a one-time batch load, it is a plain script with editable constants at the top, not an API endpoint:

# Params: edit these, then run `uv run python -m ingest.load_source_documents`
MARKDOWN_DIR = Path(__file__).resolve().parents[2] / "data" / "markdown"
SKIP_EXISTING = True

fields = {
    "ticker": filing["ticker"],
    "company_name": COMPANY_NAMES.get(filing["ticker"]),
    "form": filing["form"],
    "filing_date": parse_date(filing["filing_date"]),
    "fiscal_year": fiscal_year_from_report_date(filing.get("report_date")),
    "accession_number": accession_number,
    "source_url": filing["source_url"],
    "markdown_content": markdown_path.read_text(encoding="utf-8"),
    "ingested_at": datetime.now(UTC),
}

uv run python -m ingest.load_source_documents

Check the table in Supabase. You should see 25 records, five companies times five years, each with its metadata and the full Markdown in one text column.

Chunk with the hybrid chunker

Naive chunking splits every N tokens regardless of content. Docling's HybridChunker does better. It applies token-aware refinements on top of hierarchical, structure-based chunking, so chunks follow the document's actual headings and never exceed the token budget. That budget is set by the embedding model you are about to use.

The setup lives in backend/ingest/chunking.py:

CHUNK_MAX_TOKENS = 512

class PatchedOpenAITokenizer(OpenAITokenizer):
    """Allow tiktoken special tokens that appear in SEC filing text."""

    def count_tokens(self, text: str) -> int:
        return len(
            self.tokenizer.encode(
                text=text,
                allowed_special=set(),
                disallowed_special=(),
            )
        )

def build_hybrid_chunker(max_tokens: int = CHUNK_MAX_TOKENS) -> HybridChunker:
    # HybridChunker applies token-aware splits on top of HierarchicalChunker output.
    return HybridChunker(
        tokenizer=build_tokenizer(max_tokens=max_tokens),
        merge_peers=True,
        repeat_table_header=True,
        serializer_provider=MarkdownTableSerializerProvider(),
    )

The tokenizer patch matters. SEC filings occasionally contain strings that cl100k_base treats as special tokens, and unpatched tiktoken raises on them mid-corpus. One subtlety came out of planning this with the agent. The chunker works on Docling's document representation, not the exported Markdown, so chunk_document re-converts each HTM file and chunks that. This preserves page numbers and section headings, which each ChunkRecord carries along:

@dataclass(frozen=True, slots=True)
class ChunkRecord:
    chunk_index: int
    text: str
    page: str | None
    section: str | None
    token_count: int
    chunk_metadata: dict[str, Any]

That metadata is what makes citations like "Apple 10-K, page 24, Item 7" possible later.

Embed the chunks

backend/ingest/embeddings.py calls OpenAI's text-embedding-3-small at 1536 dimensions (matching the vector(1536) column from the migration), batching 100 texts per request and validating the dimension of every vector that comes back.

Embeddings cost real money, so do not start with the full corpus. Dave's instruction to the planning agent was "we first want to test it with one chunk, upload it to the database, so we make sure everything works before running everything." The CLI supports exactly that:

# test first: one document, one chunk, no writes
uv run python -m ingest.chunk_and_embed --accession 0000320193-23-000106 --max-chunks 1 --dry-run

# then the full corpus
uv run python -m ingest.chunk_and_embed --all

The full 25-filing corpus came to about 11 cents of embedding spend. Small, but the habit of testing on one chunk first is what keeps a bug from costing you the full run several times over.

An idempotent CLI

backend/ingest/chunk_and_embed.py is built to be re-run safely. It skips documents that already have chunks unless you say otherwise:

target = parser.add_mutually_exclusive_group(required=True)
target.add_argument("--accession", help="Process one filing by accession number")
target.add_argument("--all", action="store_true", help="Process all manifest filings")
parser.add_argument("--dry-run", action="store_true",
                    help="Chunk only; no embeddings or database writes")
parser.add_argument("--force", action="store_true",
                    help="Delete existing chunks for target document(s) before re-ingesting")

So a partial failure is cheap to recover from. Re-run --all and finished documents are skipped, or re-run one filing with --accession ... --force to wipe and rebuild just its chunks. The generated tsvector column populates itself on insert, so full-text search needs no extra step.

A known weakness, flagged for later

Inspecting the fresh chunks, Dave immediately flags a problem, "there is table data in here, but it has very little context. For pure semantic search, this is not going to be that good." A chunk containing a bare slice of numbers does not embed into anything a question like "what was Apple's product revenue" will land near.

He leaves it deliberately, "optimizing how you get your chunks in the database is one of those things where you can spend an entire week, sometimes a month, per project." The pipeline works end to end; the table problem gets a dedicated fix in the table extraction stage.

Checkpoint

Before building retrieval on top of this corpus, verify:

• source_documents has 25 rows with ticker, form, filing date, and full Markdown content
• document_chunks has rows with text, embedding, token_count at or under 512, page, and section
• The search_vector column is populated
• Spot-check a known passage by finding a chunk from Apple's revenue discussion and confirming the text, page, and section metadata match the original filing