Score the answers
Read · 1 of 2
Vibes are not a test
Changed a prompt and it "feels better"? Maybe. Or maybe it's worse on the cases you didn't try. Agents need tests, just like code.
An eval is a fixed set of tasks with known good answers, and a scorer that runs them all and reports a number. Run it before and after every change. Start with 20 real cases; that alone beats guessing.
Score with code wherever you can, and normalise first: "14", "14.0" and " 14. " are the same answer.
Read this lesson as text
A fixed set of questions with known answers, scored by code.
Vibes are not a test
Changed a prompt and it "feels better"? Maybe. Or maybe it's worse on the cases you didn't try. Agents need tests, just like code.
An eval is a fixed set of tasks with known good answers, and a scorer that runs them all and reports a number. Run it before and after every change. Start with 20 real cases; that alone beats guessing.
Score with code wherever you can, and normalise first: "14", "14.0" and " 14. " are the same answer.
Exercise: An answer scorer
Input: n, then n lines expected|got.
Two answers match if, after lower-casing, trimming spaces and a trailing ., they're equal text, or both are numbers with the same value.
Print case i: expected X, got Y for each mismatch (original text), then score: a/n (p%) with p rounded down.