Receipt Parser Engineer

Overview

Build and maintain vendor parsers that turn extracted receipt text into ReceiptNormalized with high accuracy, backed by golden fixtures. Keep parsing deterministic where possible and treat Claude Vision as the safety net for unknown vendors.

Workflow Decision Tree

1. Start from the file type

   • PDF: use embedded text when possible; if the PDF has little/no embedded text, OCR it.
   • Image: OCR it.
   • Email HTML/text: parse directly (no OCR).

2. Decide whether to write/extend a deterministic parser

   • Write/refine a deterministic parser when the vendor is high-volume, has structured line items, or needs reliable tax/subtotal/total.
   • Prefer Claude Vision fallback when the vendor is low-volume, highly variable, or you lack enough samples to stabilize patterns.

3. Choose the OCR/text-extraction path

   • Rule: pdfplumber is for text-based PDFs; AWS Textract is for images and anything pdfplumber can’t extract meaningfully from a PDF.

Core Invariants

• Check main before you build. A task worktree can be hundreds of commits behind main, and main may already have its own (different) version of this parser. Diff your base against main and check what the running worker actually has first — then port onto current main, never clobber it. See references/deploy-and-isolated-testing.md.
• Do not introduce any local OCR-binary wrapper or dependency; use Textract for OCR.
• Keep vendor parsers deterministic: parse from ocr_text where possible. Use pdf_path + word coordinates when the layout wraps multi-column data and flattened text truncates SKUs/fields (see references/sourcing-and-coordinate-parsing.md).
• Every parser change ships with a golden fixture test that reproduces the bug and prevents regressions.
• Avoid “magic balancing” lines: prefer validation_warnings for missing/faded items rather than inventing data.

Workflow: Add a New Vendor Parser (Deterministic)

1. Collect samples (they're scattered)

   • A vendor's invoices live across worktrees, drive dumps, the runtime DB, and the object store. Find + hash-dedup + layout-tag them in one shot:
     • python3 skills/receipt-parser-engineer/scripts/parser_workbench.py --discover <stem>
   • Target 3–10 real invoices spanning every layout the vendor has used over time (vendors change formats across years) plus refunds/discounts. Group by the printed fingerprint.
   • Full source map + runtime/DB queries: references/sourcing-and-coordinate-parsing.md.

2. Generate OCR text for fixtures

   • Use the OCR factory (pdfplumber → Textract fallback) from the worker container:
     • docker compose exec worker python scripts/test_vendor_parsers.py /path/to/receipt.pdf
   • Save to tests/fixtures/golden_receipts/<vendor>/<name>_ocr.txt.

3. Create expected outputs (verify against the document — don't copy the parser)

   • Fill tests/fixtures/golden_receipts/<vendor>/<name>_expected.json from the printed invoice / a vision read. Never paste the current parser's output as "expected" — the test then just asserts the bug back at itself.
   • Keep it minimal: the fields you want stable (totals, date, invoice #, line count, a few representative SKUs incl. any that wrap).

4. Implement the parser

   • Add packages/invoice_parsers/vendors/<vendor>_parser.py:
     • detect_format(ocr_text) -> bool should be strict enough to avoid false positives.
     • parse(ocr_text, entity, pdf_path=...) -> ReceiptNormalized should be resilient to OCR noise.
   • Layout dictates the technique: flat single-line rows → regex on ocr_text. Wrapped multi-column grids where flattened text truncates SKUs/fields → parse from pdf_path using word coordinates (not extract_tables; these grids often have no cell borders). See references/sourcing-and-coordinate-parsing.md.

5. Register the parser

   • Update packages/invoice_parsers/vendor_dispatcher.py:
     • import your parser
     • add it to the parser list (before GenericParser)
     • add the vendor key to GOLDEN_VENDORS when it has golden coverage

6. Add golden tests

   • Update tests/unit/test_invoice_parsers.py with a new @pytest.mark.golden class for the vendor.
   • Assert at minimum: vendor_guess, totals, purchase date, invoice number (if applicable), and line count.

7. Run tests

   • make test-golden
   • If you touch shared parsing behavior, run make test-unit too.

Workflow: Deploy a parser to runtime

The worker bind-mounts the control-plane checkout (/home/clarencehub/curlys-books), not task worktrees — so a parser is only "in runtime" once it's on main there and the worker is restarted.

1. Port your change onto current main (branch off main in the control-plane checkout).
2. Run golden tests against a throwaway Postgres (never prod) — recipe in references/deploy-and-isolated-testing.md.
3. Commit, merge to main, docker restart curlys-books-worker.
4. Verify the live process, not just the file: exec the parser inside the running worker on a sample and confirm totals reconcile / 0 truncated SKUs.
5. To fix receipts the old parser already mis-persisted, use the live-apply lane (pre-posting only) — see the deploy reference.

Workflow: Fix/Refine an Existing Parser

1. Add a failing fixture first (new <name>_ocr.txt + <name>_expected.json).
2. Make the smallest parser change that fixes the issue.
3. Re-run make test-golden until green.
4. Add/adjust validation_warnings when the receipt can’t be made internally consistent.

Debugging Playbook

• What did the live pipeline actually produce? Diff the on-disk handoff against the PDF — /srv/curlys-books/objects/{entity}/{receipt_id}/pipeline_handoffs/parse_to_persist_*.json has every line's sku/pack_count/unit_size/account/tax. This is usually the fastest way to see what the parser got wrong.

• Run the validation heuristics on any sample before trusting a parse: Σ line_total == subtotal, tax = total − subtotal, 0 truncated SKUs (see references/sourcing-and-coordinate-parsing.md).

• One-shot OCR + parse (local workbench):

  • python3 skills/receipt-parser-engineer/scripts/parser_workbench.py --file /path/to/receipt.pdf --entity corp
  • python3 skills/receipt-parser-engineer/scripts/parser_workbench.py --ocr-text tests/fixtures/golden_receipts/<vendor>/<name>_ocr.txt --entity corp

• OCR routing: packages/parsers/ocr/factory.py

• Textract provider: packages/parsers/ocr/provider_textract.py

• PDF embedded text extraction: packages/parsers/ocr/pdf_text_extractor.py

• Dispatcher + golden vendor list: packages/invoice_parsers/vendor_dispatcher.py

• Worker parse routing (incl. Claude Vision heuristic): services/worker/tasks/pipeline/parse.py

• Golden fixtures: tests/fixtures/golden_receipts/

References (load as needed)

• Sourcing scattered invoices + coordinate parsing + validation heuristics: references/sourcing-and-coordinate-parsing.md
• Deploy to runtime, isolated-worktree testing, the stale-main trap: references/deploy-and-isolated-testing.md
• Agent prompt (Claude/Codex): references/agent-prompt.md
• Repo/pipeline map: references/pipeline-map.md