Flaky agents, honest numbers
Read · 1 of 2
Same task, different results
Agents aren't deterministic: the same task can pass on Monday and fail on Tuesday. One run tells you little.
Run each task k times and report three numbers. pass@1: how often a single run succeeds. pass@k: how many tasks succeed at least once in k tries (good when a person picks the best). pass^k: how many succeed every single time (what you need when it runs unattended).
Read this lesson as text
Run each task several times. Measure how often, not just whether.
Same task, different results
Agents aren't deterministic: the same task can pass on Monday and fail on Tuesday. One run tells you little.
Run each task k times and report three numbers. pass@1: how often a single run succeeds. pass@k: how many tasks succeed at least once in k tries (good when a person picks the best). pass^k: how many succeed every single time (what you need when it runs unattended).
Exercise: Three numbers
Input: n, then n lines of results, one per task, like PPF (P pass, F fail); every line has the same length k.
Print pass@1=X% pass@k=Y% pass^k=Z% (with the real k, one decimal each).