Grade the agent
Read · 1 of 3
Does it actually work?
Four tests passing doesn't mean your agent is good. Real students ask things in their own words.
Write an eval: questions with the document that should be cited (or none if the handbook doesn't cover it), run the agent on each, and count. You're about to find failures your tests never showed.
Read this lesson as text
Run it on questions with known answers. Find out where it really fails.
Does it actually work?
Four tests passing doesn't mean your agent is good. Real students ask things in their own words.
Write an eval: questions with the document that should be cited (or none if the handbook doesn't cover it), run the agent on each, and count. You're about to find failures your tests never showed.
Exercise: The eval
Your agent from the last lesson runs first (open Runs first to see it), as run_agent(question, log=False).
Input: n, then n lines expected|question, where expected is a document id or none.
Run the agent on each. The citation is the [id] at the end of the answer, or none if there isn't one. Print PASS <question> or FAIL <question> (cited X, expected Y), then score: a/n.
What the failures tell you
"room" didn't match "rooms", and "marks" didn't match "results". Word counting can't see that they mean the same thing. That's precisely what embeddings fix: they place words with similar meanings close together, so "marks" lands near "results".
This is how real agent work goes: build it, eval it, read the failures, fix the weakest part, eval again. Swap in a real embedding model and a real Claude model (module 12), and this help desk would be ready to try with real students.
You've built an agent from an empty file: the loop, tools, structured output, context, retrieval, planning, reliability, safety, evals and teams. Now take the final exam.