I failed an agent trajectory eval round

A quick note first. The candidate in this story is an illustrative composite, not a real person, and the interview is not a report of any real company's process or result. It is a realistic pattern, told in first person so the mistakes are easy to recognize.

The round was about evaluating an agent. The interviewer shared a log of an agent handling a customer request: it read the ticket, looked up an order, checked a policy, issued a partial refund, and replied to the customer. The question was simple. Is this a good run, and how would you evaluate thousands of runs like it? I read the final reply, saw that it was polite and correct, and said yes. That was the first of several mistakes.

Mistake one was grading the reply instead of the trajectory. The reply said the refund was done. The interviewer asked me to check the steps. The agent had looked up the right order, but it had called the refund tool twice, and the second call had refunded a different amount. The reply looked fine because the model only reported the first result. What I would do now is treat the trajectory, meaning every step the agent took, and the end state it left behind as the real output. The reply is one piece of evidence, not the verdict.

Mistake two was having no definition of a good path. When the interviewer asked what a correct trajectory looks like, I described this one run. I had no general answer. What I would do now is define, for each task type, what must happen, what must never happen, and what is allowed to vary. A refund task must check the order and the policy and change exactly one record. It must never refund twice. The order of the lookups can vary.

Mistake three was only checking the final state. Once I understood the double refund, I swung too far and said I would just compare the database before and after. The interviewer pointed out that some bad paths end in the right state by luck, for example reading data the agent was not authorized to see before doing the right thing. What I would do now is check both: the end state for correctness, and the trajectory for things like permission checks, unnecessary tool calls, loops, and steps that touched data they should not have.

Mistake four was trusting a single run. I evaluated one log. The interviewer asked whether the agent would do the same thing next time. Agents can take different paths on the same input, so one good run proves little. What I would do now is run each task several times and report how consistently it succeeds, not just whether it succeeded once.

Mistake five was vague scoring. Asked how I would score trajectories at scale, I said I would use a model as a judge. The follow-up was how I would know the judge was right. I had no answer. What I would do now is use deterministic checks wherever possible, such as whether a tool was called, whether a record changed, or whether a step limit was hit, and use a model judge only for the parts that need judgment, with a small set of human-labeled runs to check the judge against.

The fix, in the end, was a simple frame I now use for any agent evaluation question. Check the end state. Check the trajectory against what must and must never happen. Run it more than once. Use code checks first and a judge second, and check the judge. The LLM evaluation guide at /blog/llm-evaluation-interview-guide, the agent tool design story at /blog/bombed-agent-tool-design-round-checklist, and the LangGraph agent system design guide at /blog/langgraph-agent-system-design-interview-guide are the reading I would pair with that frame.