Four ACORD form types, fully synthetic training data, commodity GPUs, and a verification culture that treated every claim as unverified until an instrument said otherwise.
Commercial-insurance submissions arrive as ACORD forms — dense multi-column grids, checkbox arrays, rating worksheets — filled by brokers and degraded by scanners. Downstream systems need exact JSON: applicant identity, FEIN, class codes and payrolls, property limits. And one requirement most extraction projects quietly skip: when a field is absent, the correct answer is an explicit null, not a plausible guess. In insurance, an invented experience-mod is worse than a missing one.
Two constraints shaped everything. The documents are sensitive, so the model had to self-host privately — no third-party APIs. And no labeled corpus existed, so training data had to be manufactured, which frames the whole project: how do you come to trust a model trained entirely on data you invented? Our answer: you don't. You build instruments that could prove it wrong, and you let them try.
Before any training, roughly a dozen iterations went into the pipeline: a profile generator (hundreds of fictional companies with coherent industries, geographies, class codes, payrolls), a document assembler (multiple layout styles, label-synonym pools, OCR-noise modes with value protection), and an evaluator scoring parse rate, per-field exact match, and whole-document exact match.
The founding lesson arrived immediately: the first pipeline printed "GOLD STANDARD ACHIEVED" over an empty output file — a silent-success bug. Everything after descends from that moment. Validators guard every stage: values verbatim in their documents, label pools collision-free, train and eval vocabularies provably disjoint, every artifact answering to a SHA-256. The rule that ran the campaign: verify effects, not intentions.
Wave 1 (3,700 documents, ~52 minutes on a rented A100) scored a perfect 1.000 on its held-out evaluation. A traditional project ships here. Instead, four stress instruments — each isolating one axis, on companies the model had never seen — told the truth: structure shift 0.503, never-trained vocabulary 0.150, raw prose 0.463, and systematic failure on absence cases. Diagnosis: a label-matcher wearing perfect holdout scores. The perfect number and the 0.150 described the same weights — the difference was whether the instrument could tell them apart.
Wave 2 rebuilt the corpus around variety: 11,400 documents mixing the original set, augmented layouts and synonyms, and an independently-generated long-form set — 7.1 GPU-hours. Structure shift solved outright (1.00); unseen vocabulary 0.150 → 0.843; prose 0.463 → 0.956; regression still perfect. Then the pre-registered gate v2 fired — the explicit-null requirement scored 0.50 whole-document; the model was filling absent fields with plausible values. The gate refused to pass it. No goalposts moved; a fix wave was scheduled.
Wave 2b trained a null-teaching booster (3,590 lines, 1.8 GPU-hours, continuing from wave 2). Gate v2.1 — five pre-registered clauses — failed 3 of 5. Our working diagnosis blamed an interaction: a sibling drill strengthening a "fields always exist" prior. Then the method's core move: regenerate every failure, classify every error. The forensics said the opposite — all 30 nulled fields were answered with clean, explicit nulls. The failing scores were label bindings: one identifier missing 28/30 (every training synonym had shared the same token — the model learned the token, not the concept), two limit bindings at 15/30 and 9/30 where the model abstained rather than guessed. Two lessons banked: the instruments falsified their designer's theory and were right; aggregates without forensics send you fixing the wrong thing.
The binding booster (2,020 lines, 1.1 GPU-hours) widened training vocabulary for exactly the three measured fields — concept-teaching synonyms sharing no tokens with the untouchable eval labels. Same frozen instruments, no threshold changes — and the climb, wave over wave:
| Gate v2.1 clause | Required | Wave 2b | Wave 2c |
|---|---|---|---|
| Explicit nulls — whole-document | ≥0.85 | 0.583 | 0.900 ✅ |
| Explicit nulls — field | ≥0.95 | 0.934 | 0.985 ✅ |
| Held-out regression | ≥0.99 | 1.000 | 1.000 ✅ ×3 |
| Unseen labels — form 130 | ≥0.85 | 0.909 | 0.994 ✅ |
| Unseen labels — form 140 | ≥0.85 | 0.817 | 1.000 ✅ |
Every instrument re-run current: 20 of 25 cells perfect across five instruments × four forms, the five imperfect cells individually diagnosed:
Then one last synthetic exam, because composition had never been tested: forty documents carrying every stressor simultaneously — never-trained vocabulary, shuffled structure, character corruption inside the values (gold = the clean value: a restoration test), absent fields, the largest worksheets — plus deliberate traps: a fake insured with a FEIN-shaped number, a nine-digit decoy aimed at the identifier field, an expiring premium beside the real limits.
Result: parse 1.0 · field 0.986 · whole-document 0.900. Forensic classification of every erroneous field: misattribution 0 — all forty traps refused, and the one failure class a source-verification guardrail cannot catch did not occur. Restoration-miss 0: every corrupted value the model emitted, it emitted repaired. Three conservative omissions, and exactly one invented value in the forty hardest documents — a digit-hallucination during restoration, precisely the class the mandatory production guardrail (every value must appear in the source) catches deterministically.
The campaign lives in a 73-file private registry — three adapters, ten datasets, every report, the diagnostic tooling, and a run log explaining every decision — with the gate-passing adapter byte-verified (LFS SHA-256). Then we killed the training machine deliberately. On a fresh GPU of a different architecture: one command restored all 73 files in 27 seconds; the dependency stack reproduced itself to the recorded digit; and the restored model matched its own behavioral fingerprint exactly — same scores to four decimals, same four hardest documents failed. The drill also surfaced a real dual-Python template hazard, permanently hardening the restore runbook. A 12-document production-simulation set (scanner stamps, fax headers, mixed vocabulary) then scored 11/12 on the restored machine — the single miss landing in the exact field, at the exact rate, the model's own limitations ledger had documented in advance.
Synthetic evaluation, however adversarial, shares one bloodline: we authored it. Everything above proves the model reads our documents. The real-data program removes that asterisk in two stages, tracked publicly on this page.
Stage 1 — Production-realism campaign (now running). Every component that can be real, is real: genuine fillable ACORD eForms for three form types (349, 491, and 356 live form fields) plus a faithfully rebuilt state-edition auto section (239 fields); a machine broker filling 28 applications with fictional insureds — correct entity checkboxes, worksheet rows that sum, realistic clutter, per-document extraction verification; an adversarial second model independently auditing every filled form, sharing no context with the filler; the customer's own OCR pipeline as the transform — the exact production path, quirks included; and gold labels written by human hand from the PDFs, gated by a four-document pilot. The fill stage produces no answer key — the human labels are the only gold, and the model never sees a byte of this data before scoring.
Filler complete and independently spot-audited · adversarial verification in progress · scoring follows hand-labeling. Thresholds and predictions are pre-registered before scoring, in the same discipline as every gate above.
Results will be published here when complete — pass or fail.
Stage 2 — Customer-paper evaluation (scheduled). The final bar: genuine scanned customer documents, anonymized, hand-labeled, evaluation-only. Same evaluator, same forensics, same publish-regardless commitment. No accuracy claim on this page graduates from "synthetic" to "production" until this stage says so.
All accuracy figures above were measured on synthetic evaluations built from held-out fictional companies. Five residual weaknesses are individually named in the model card — all conservative in character: under extreme stress the model abstains or omits; it does not invent (one guardrail-catchable exception across forty maximally-stressed documents).
Real-data results are pending per Phase VIII. Production deployment mandates the value-in-source guardrail regardless of any score.
A 14B model, tens of dollars of compute, and zero real training documents reached production-candidate accuracy — because every claim was treated as unverified until an instrument said otherwise, including the ones we most wanted to believe. The model is the artifact. The method is the product — and its final clause is running right now, on this page.
A 30-minute model audit maps your extraction problem to the same discipline you just read — instruments, gates, and forensics included. If fine-tuning isn't the answer, we'll say that too.
Book my free model auditno retainer · baseline guarantee on every build
Read next → Fine-tuning a 4B legal clause analyzer to 98.7%
DUC