Make Agents Better
When you change an agent, you must know if the change made it better or worse. In this track, you read a run of the pr-review agent step by step, and you set limits on its time and cost. Then you write an eval that checks the review of a pull request with four known bugs, and you compare two versions of the agent. Last, you review the memories that agents find and use.
| Page | You do | You learn |
|---|---|---|
| 1. Read a run like a debugger | Follow the tool calls of a run. Make a run fail on its model request limit. Find the cost of a run. | The Transcript, the Limits card, limits and timeout, and Insights |
| 2. Write an eval | Write an eval that checks for four known bugs. Score runs, change the instructions, and compare the scores. | Eval files, rubrics, judges, scores and the Evals tab |
| 3. Review what agents remember | Make an agent propose a memory. Approve it, see the review agent use it, and archive it. | Memories, the review queue, create_memory and search_memory |
The track takes about 45 minutes.
Before you start
- Do the Quick Start, Build Your First Agent chapters 1 to 3, and steps 1 and 3 of chapter 4. The agent must clone the repository and have the read-only tool filters. These steps need no GitHub Integration Connection. You need the
pr-reviewagent and thereview-rulesskill in your project. - Use the example pull request autohealai/astronomy-shop-template#1. It adds a timeout to the call from the recommendation service to the product catalog. The repository is public, so you do not need a GitHub Integration Connection.
- You need the Member role, to commit to the project, start runs, score runs and review memories.
- For the "From the CLI" sections, install the
autohealCLI and log in. See chapter 8. - The pages change
pr-review. To keep your tutorial agent as it is, do the track on a branch. See chapter 6.
Keep post_comment off in every run in this track.