3. Return Structured Results and Artifacts
Until now, the agent returns the review as text. A person can read text, but a script cannot easily find the verdict in it. In this page, the review step returns named fields: verdict, blockers, summary and the ID of an artifact. The agent passes those fields on as its own result. The full review goes into an artifact, a document that Autoheal keeps after the run. At the end, a second agent runs pr-review-steps and reads its verdict.
Before you start
Do page 2 first.
Step 1: Add an artifact template
An artifact template tells the agent what a type of document must contain. Each template that an agent names gives it a tool named create_artifact_<template id>.
In the editor, create the folders artifact-templates/review in agents/pr-review-steps, and create the file TEMPLATE.md in it:
---
id: review
name: Pull request review
description: The review of one pull request, with its verdict, findings and checklist.
format: text/markdown
---
Write the review in Markdown with these sections:
1. **Verdict**: merge, merge with changes, or do not merge, and one sentence that says why.
2. **Findings**: one line for each finding. Give the severity, the file and line, the problem and the fix.
3. **Checklist**: each review rule, with pass or fail.
4. **Question for the author**.
Give the artifact the title "Review of <repository>#<number>".
The folder name is the template's ID. It must be lowercase letters, digits and hyphens. A template in artifact-templates/ at the root of the project is for all agents. A template in an agent's own folder is for that agent only. To check the content against a JSON Schema, or to show it as a designed page, see Artifact Templates.
Step 2: Make the review step return fields
In agent.yaml, change the system_tools field of the review step, and add artifacts and returns below integration_tools:
system_tools: operation_type == "read" || name == "create_artifact"
integration_tools: integration_type == "github" && operation_type == "read"
artifacts:
- ./artifact-templates:review
returns:
verdict:
type: string
enum: [merge, merge_with_changes, do_not_merge]
required: true
blockers:
type: array
description: One line for each Blocker finding. Empty when there are none.
required: true
items: string
summary: {type: string, required: true}
review_artifact_id: string
| Field | What it does |
|---|---|
system_tools | Keeps create_artifact. Without it, the agent cannot use the template tool, and the push gives a warning. |
artifacts | ./artifact-templates:review is the template in this agent's folder. It gives the step the tool create_artifact_review. |
returns | The fields of the step's result. enum limits verdict to three values. A field is optional unless it has required: true. |
Add two items at the end of the numbered list in instructions.md:
8. Save the full review with `create_artifact_review`.
9. Return the verdict, each Blocker finding, a summary of two or three sentences, and the ID of the artifact.
Step 3: Give the agent outputs
The review step is inside the if step from page 1. A step outside size cannot name review. It reads steps.size.output.steps.review. That value is empty when the run took else. Add this outputs field at the end of agent.yaml, at the top level:
outputs:
verdict: ${{ steps.size.output.steps.review?.verdict ?? "not_reviewed" }}
blockers: ${{ steps.size.output.steps.review?.blockers ?? [] }}
summary: ${{ steps.size.output.steps.review?.summary ?? steps.size.output.steps.too_large.message }}
review_artifact_id: ${{ steps.size.output.steps.review?.review_artifact_id ?? "" }}
?. gives an empty value when review did not run, and ?? then gives the value on its right. The agent's result now has the same four fields after each run, on both branches.
Step 4: Commit and run
- Commit and run
Commit the 3 files. Run the agent on
https://github.com/autohealai/astronomy-shop-template/pull/1. - Read the result
Open the Summary tab. Check that it shows the summary in two or three sentences. Under Artifacts, check that there is a card named "Review of autohealai/astronomy-shop-template#1". Click it to read the full review.
- Find the tool calls
In the Transcript tab, open Review the diff. Check that the step called
create_artifact_reviewbefore it returned. - Run the else branch
Run the agent again with max_lines set to
10. Check that the run succeeds, and that the summary is the message of thetoo_largestep.
The artifact stays after the run. You also find it on the agent's Artifacts tab, and in Work → Artifacts.
Step 5: Read the fields from the CLI
autoheal run pr-review-steps "https://github.com/autohealai/astronomy-shop-template/pull/1" --wait -o json > run.json
jq '.output.result.verdict' run.json
jq '.output.result.blockers' run.json
autoheal artifacts get "$(jq -r '.output.result.review_artifact_id' run.json)" --include content
Check that verdict is merge, merge_with_changes or do_not_merge, and that blockers is a list. With max_lines set to 10, verdict is not_reviewed and blockers is [].
Step 6: Read the fields from another agent
An agent step runs another agent of the project, and its output is that agent's result. In the editor, click New → New agent and name it merge-gate. Replace its agent.yaml with this:
apiVersion: agents.autoheal.ai/v1
name: Merge gate
description: Runs pr-review-steps on one pull request and fails when the verdict is do_not_merge.
inputs:
prompt:
type: string
required: true
description: The pull request URL.
steps:
- id: review
agent: agents:pr-review-steps
with:
prompt: ${{ inputs.prompt }}
- id: gate
runtime: bash
env:
- name: VERDICT
value: ${{ steps.review.output.verdict }}
- name: BLOCKERS
value: ${{ steps.review.output.blockers }}
code: |
echo "Verdict: $VERDICT" >&2
printf '{"verdict": "%s"}' "$VERDICT" > "$AUTOHEAL_OUTPUT"
if [ "$VERDICT" = "do_not_merge" ]; then
echo "Blockers: $BLOCKERS" >&2
exit 1
fi
Commit, and run merge-gate on the example pull request. When the verdict is do_not_merge, the gate step fails, and the run fails. Check that the error of the gate step shows the verdict and the list of blockers. A list or an object in env arrives as JSON text.
A trigger starts an agent, but it does not read the agent's result. To act on the verdict of a review that a trigger starts, make the trigger start merge-gate instead of pr-review-steps.
What to know
- In
returns, writerequired: trueon each field. Do not write a JSON Schemarequired: [a, b]list. For an array, mark the array field, not itsitems. See Nested objects and arrays. returnschanges only the run's result. A follow-up message in the run's chat answers in text.- A step in
thenorelsecannot be read by its own ID from outside theifstep.${{ steps.review.output.verdict }}inoutputspasses the push, but the run fails withcannot fetch output from <nil>. Readsteps.size.output.steps.reviewinstead. - Without
?.and??, an output that reads the branch that did not run fails the run, for example withcannot fetch verdict from <nil>. - When an agent has
outputs, its result is theoutputsobject. Withoutoutputs, its result is the output of its last step. On page 1, that was theifstep's{branch, steps}object. - When a code step exits with a code that is not 0, the step's error is what the script wrote to stderr. The step has no output, even if it wrote
$AUTOHEAL_OUTPUT. agent: agents:pr-review-stepsruns the agent in its own sandbox. It does not see the files of the step that calls it.
From the CLI
autoheal agents push agents/pr-review-steps -m "Return a verdict and save the review as an artifact"
autoheal run pr-review-steps "https://github.com/autohealai/astronomy-shop-template/pull/1" --commit <sha> --wait -o json
autoheal agents push agents/merge-gate -m "Fail on a do_not_merge verdict"
autoheal run merge-gate "https://github.com/autohealai/astronomy-shop-template/pull/1" --commit <sha> --wait -o json
autoheal agents push pushes the whole agent folder, the artifact template included.
See Structured results, Step outputs and Calling agents and tools in the reference.