Check the path, not just the answer
Read · 1 of 2
How it got there matters
An agent can land on the right answer by luck: guessing instead of looking it up, or calling a dangerous tool along the way.
So also check the trajectory: the list of tool calls. Did it search before answering? Did it avoid forbidden tools? Did it finish within a sensible number of steps?
Read this lesson as text
A right answer reached the wrong way is a bug waiting to happen.
How it got there matters
An agent can land on the right answer by luck: guessing instead of looking it up, or calling a dangerous tool along the way.
So also check the trajectory: the list of tool calls. Did it search before answering? Did it avoid forbidden tools? Did it finish within a sensible number of steps?
Exercise: Check a run
Input: the most tool calls allowed, then the run's calls in order (space-separated tool names, ending with answer). answer itself doesn't count as a tool call.
Rules: search must come before answer; delete_file is never allowed; no more than the allowed number of tool calls.
Print pass, or each broken rule: answered without searching, called delete_file, too many calls (N > M).