Evaluation Strategy trade-offs for a document intelligence pipeline

Quick answer

For a document intelligence pipeline, treat the choice as timing vs durability: a fast heuristic eval (spot checks, regex/field rules, small human samples) ships in weeks and catches obvious extraction and routing bugs, but it under-measures layout drift, long-tail docs, and model/prompt regressions. A platform refactor (versioned golden sets, automated parse/extract/classify scores, CI gates, slice dashboards) takes a quarter but becomes the release brake you need. Over two quarters, do both in sequence: Q1 ship heuristic gates plus a thin golden set on the top failure modes