Skip to main content

1. Read a Run Like a Debugger

A review can be wrong for many reasons. The agent read the wrong files, it did not load the skill, or it stopped early. The Summary shows only the answer. The Transcript and the run's panel show how the agent got the answer, and what the run used. In this page, you read a run step by step, you make a run fail on purpose, and you find the cost of a run.

Step 1: Run the agent​

  1. Open the agent in Agent Studio

    Go to Agent Studio → Agents and open pr-review. Open the agent from Agent Studio, not from Work. Only the Agent Studio view of a run shows the Limits and Evals cards.

  2. Start a run

    On the Overview tab, paste https://github.com/autohealai/astronomy-shop-template/pull/1, keep post_comment off, and click Start run.

  3. Wait for the run to end

    The run takes about 30 seconds. Check that its status is Succeeded.

Step 2: Follow the tool calls in the Transcript​

Open the Transcript tab. Each tool call shows its arguments and its result. Read the calls in order and compare them with the instructions:

Tool callWhat it tells you
load_skill with skills:review-rulesThe agent loaded the team's review rules before it started.
git_clone with the ref pull/1/headThe agent cloned the repository at the pull request.
bash with git diff origin/HEAD...HEADThe agent compared the pull request with the default branch.
search_memoryThe agent looked for lessons from earlier runs. See page 3.
bash with cat -n on a changed fileThe agent read the changed files in full, as step 3 of its instructions says.

Check that the agent did each step of its instructions. When a review is wrong, find the first tool call that does not match the instructions. That call is usually where you must change the instructions or a skill.

Step 3: Read the run's panel​

The panel beside the run has these cards:

CardWhat it shows
OverviewStatus, Agent, Started, Total duration, Branch and Started by
LimitsTime limit, Model requests, New tokens and Cost, each with what the run used
EvalsThe scores of the run. See page 2.
MemoryThe memories that agents proposed from this run

In the Limits card, Model requests shows the requests that the run made against the limit in agent.yaml, for example 6 of 20. Cost is an estimate at the model's public list price.

Step 4: Make a run fail on its limit​

In agent.yaml, change the model request limit to 3:

YAML
limits:
model_requests: 3

Commit, and run the agent with the same URL. The review needs more than 3 model requests, so the run stops. Check these items:

  • The status of the run is Failed.
  • The panel shows This run failed and the reason: step "main" failed: model request limit of 3 reached.

Change model_requests back to 20 and commit.

Step 5: Set cost and token targets​

Add a cost target and a token target below model_requests:

YAML
limits:
model_requests: 20
cost_usd: 0.5
tokens: 200000

Commit, and run the agent again. The run succeeds. In the Limits card, check that Cost and New tokens show what the run used against each target.

These targets do not stop a run. Before each model request, the agent sees what it used against each target, and it uses that to plan its work. Only model_requests and timeout stop a run:

FieldStops the runDefaultMaximum
timeoutYes10m1h
limits.model_requestsYes3001000
limits.cost_usdNo. It is a target.NoneNone
limits.tokensNo. It is a target.NoneNone

Step 6: Find the cost of many runs​

Go to Insights. The page shows the runs of your project in a time window:

  • The Spend section shows Est. spend for all runs, with input, output and cache read tokens, and Spend per day.
  • In the Runs and agents section, the Agents table has a row for each agent. Find pr-review, and read its Est. spend, Cost per run and Evaluated columns.

Use Cost per run to see the effect of a change, for example a new model or new instructions.

What to know​

  • A run without a usage report is not in the spend totals. The Limits card then shows "Usage was not recorded for this run."
  • Cost and new tokens include the subagents that the agent calls.
  • A very short timeout can stop a run before it starts. We tried timeout: 10s. The run failed at once with this error: the calling run has 9s left, too little to start this agent (it needs at least 30s plus 1m0s to use the answer). Give a single-step agent at least 2m.
  • Work → Runs opens the same run without the Limits and Evals cards. Open runs from the agent's Runs tab in Agent Studio when you debug.

From the CLI​

BASH
autoheal run pr-review "https://github.com/autohealai/astronomy-shop-template/pull/1" --follow
autoheal runs list --agent pr-review --limit 3
autoheal runs view <run ID>

--follow prints each tool call as the run makes it. autoheal runs view shows the status, the commit and the result of a run. The CLI does not show the cost of a run. Use the run's panel or Insights for cost.

See Timeout and limits in the reference, and Observe for how Autoheal records each run.