Design Evaluation Strategy architecture for a coding copilot for a large engineering org

Quick answer

Treat evaluation as its own production service next to the coding copilot, not a one-off notebook. Split the stack into: (1) corpus/index eval for the ~50M indexed docs (freshness, coverage by language/repo, retrieval hit quality), (2) offline suggestion eval on versioned golden tasks (accept/edit/reject labels, unit-test pass where applicable), and (3) online telemetry (accept rate, edit distance, revert/undo, latency, cost) sliced by language, task type, and org. Gate model, prompt, and index changes with CI on golden sets, canary by cohort, and explicit rollback when any critical slice drops even if the global score looks fine. Prototype eval that only spot-checks happy-path Python will not scale; you need stratified fixtures, owners, and release brakes.