Skip to main content

Make Agents Better

When you change an agent, you must know if the change made it better or worse. In this track, you read a run of the pr-review agent step by step, and you set limits on its time and cost. Then you write an eval that checks the review of a pull request with four known bugs, and you compare two versions of the agent. Last, you review the memories that agents find and use.

PageYou doYou learn
1. Read a run like a debuggerFollow the tool calls of a run. Make a run fail on its model request limit. Find the cost of a run.The Transcript, the Limits card, limits and timeout, and Insights
2. Write an evalWrite an eval that checks for four known bugs. Score runs, change the instructions, and compare the scores.Eval files, rubrics, judges, scores and the Evals tab
3. Review what agents rememberMake an agent propose a memory. Approve it, see the review agent use it, and archive it.Memories, the review queue, create_memory and search_memory

The track takes about 45 minutes.

Before you start​

  • Do the Quick Start, Build Your First Agent chapters 1 to 3, and steps 1 and 3 of chapter 4. The agent must clone the repository and have the read-only tool filters. These steps need no GitHub Integration Connection. You need the pr-review agent and the review-rules skill in your project.
  • Use the example pull request autohealai/astronomy-shop-template#1. It adds a timeout to the call from the recommendation service to the product catalog. The repository is public, so you do not need a GitHub Integration Connection.
  • You need the Member role, to commit to the project, start runs, score runs and review memories.
  • For the "From the CLI" sections, install the autoheal CLI and log in. See chapter 8.
  • The pages change pr-review. To keep your tutorial agent as it is, do the track on a branch. See chapter 6.

Keep post_comment off in every run in this track.