Skip to content

Check the path, not just the answer

Read · 1 of 2

How it got there matters

An agent can land on the right answer by luck: guessing instead of looking it up, or calling a dangerous tool along the way.

So also check the trajectory: the list of tool calls. Did it search before answering? Did it avoid forbidden tools? Did it finish within a sensible number of steps?

Read this lesson as text

A right answer reached the wrong way is a bug waiting to happen.

How it got there matters

An agent can land on the right answer by luck: guessing instead of looking it up, or calling a dangerous tool along the way.

So also check the trajectory: the list of tool calls. Did it search before answering? Did it avoid forbidden tools? Did it finish within a sensible number of steps?

Exercise: Check a run

Input: the most tool calls allowed, then the run's calls in order (space-separated tool names, ending with answer). answer itself doesn't count as a tool call.

Rules: search must come before answer; delete_file is never allowed; no more than the allowed number of tool calls.

Print pass, or each broken rule: answered without searching, called delete_file, too many calls (N > M).