Agent Evaluation

Overview

Agents in production need fixes and improvements to sustain the test of time. The only way to do this with enough guarantees is by monitoring the agent and by building robust benchmarks that measure its performance. Learn how to design that benchmark, judge its results and follow an agent through its traces. Agentic AI Foundations uses a proxy to see what a harness sends to the model, while this course uses traces to evaluate and debug the agent. In the hands-on parts you build a benchmark for an agent and use its traces to find out why it fails.

What you'll cover

• ▸Why agents are harder to evaluate than normal software
• ▸How to design a test set: what to measure and where test cases come from
• ▸The typical mistakes agents make: poor context, getting lost in long tasks and calling the wrong tool
• ▸How to judge results: code, LLM or human
• ▸What a benchmark needs underneath it: tracing, versioning, snapshots
• ▸Benchmarking tools compared: Phoenix, Harbor and DeepEval, and when to use which

What you'll be able to do

• ✓Design and implement a benchmark for an agent
• ✓Follow an agent's execution end to end through its traces
• ✓Judge whether a new version is better or worse