Skip to main content

3. Return Structured Results and Artifacts

Until now, the agent returns the review as text. A person can read text, but a script cannot easily find the verdict in it. In this page, the review step returns named fields: verdict, blockers, summary and the ID of an artifact. The agent passes those fields on as its own result. The full review goes into an artifact, a document that Autoheal keeps after the run. At the end, a second agent runs pr-review-steps and reads its verdict.

Before you start​

Do page 2 first.

Step 1: Add an artifact template​

An artifact template tells the agent what a type of document must contain. Each template that an agent names gives it a tool named create_artifact_<template id>.

In the editor, create the folders artifact-templates/review in agents/pr-review-steps, and create the file TEMPLATE.md in it:

MARKDOWN
---
id: review
name: Pull request review
description: The review of one pull request, with its verdict, findings and checklist.
format: text/markdown
---

Write the review in Markdown with these sections:

1. **Verdict**: merge, merge with changes, or do not merge, and one sentence that says why.
2. **Findings**: one line for each finding. Give the severity, the file and line, the problem and the fix.
3. **Checklist**: each review rule, with pass or fail.
4. **Question for the author**.

Give the artifact the title "Review of <repository>#<number>".

The folder name is the template's ID. It must be lowercase letters, digits and hyphens. A template in artifact-templates/ at the root of the project is for all agents. A template in an agent's own folder is for that agent only. To check the content against a JSON Schema, or to show it as a designed page, see Artifact Templates.

Step 2: Make the review step return fields​

In agent.yaml, change the system_tools field of the review step, and add artifacts and returns below integration_tools:

YAML
        system_tools: operation_type == "read" || name == "create_artifact"
integration_tools: integration_type == "github" && operation_type == "read"
artifacts:
- ./artifact-templates:review
returns:
verdict:
type: string
enum: [merge, merge_with_changes, do_not_merge]
required: true
blockers:
type: array
description: One line for each Blocker finding. Empty when there are none.
required: true
items: string
summary: {type: string, required: true}
review_artifact_id: string
FieldWhat it does
system_toolsKeeps create_artifact. Without it, the agent cannot use the template tool, and the push gives a warning.
artifacts./artifact-templates:review is the template in this agent's folder. It gives the step the tool create_artifact_review.
returnsThe fields of the step's result. enum limits verdict to three values. A field is optional unless it has required: true.

Add two items at the end of the numbered list in instructions.md:

MARKDOWN
8. Save the full review with `create_artifact_review`.
9. Return the verdict, each Blocker finding, a summary of two or three sentences, and the ID of the artifact.

Step 3: Give the agent outputs​

The review step is inside the if step from page 1. A step outside size cannot name review. It reads steps.size.output.steps.review. That value is empty when the run took else. Add this outputs field at the end of agent.yaml, at the top level:

YAML
outputs:
verdict: ${{ steps.size.output.steps.review?.verdict ?? "not_reviewed" }}
blockers: ${{ steps.size.output.steps.review?.blockers ?? [] }}
summary: ${{ steps.size.output.steps.review?.summary ?? steps.size.output.steps.too_large.message }}
review_artifact_id: ${{ steps.size.output.steps.review?.review_artifact_id ?? "" }}

?. gives an empty value when review did not run, and ?? then gives the value on its right. The agent's result now has the same four fields after each run, on both branches.

Step 4: Commit and run​

  1. Commit and run

    Commit the 3 files. Run the agent on https://github.com/autohealai/astronomy-shop-template/pull/1.

  2. Read the result

    Open the Summary tab. Check that it shows the summary in two or three sentences. Under Artifacts, check that there is a card named "Review of autohealai/astronomy-shop-template#1". Click it to read the full review.

  3. Find the tool calls

    In the Transcript tab, open Review the diff. Check that the step called create_artifact_review before it returned.

  4. Run the else branch

    Run the agent again with max_lines set to 10. Check that the run succeeds, and that the summary is the message of the too_large step.

The artifact stays after the run. You also find it on the agent's Artifacts tab, and in Work → Artifacts.

Step 5: Read the fields from the CLI​

BASH
autoheal run pr-review-steps "https://github.com/autohealai/astronomy-shop-template/pull/1" --wait -o json > run.json
jq '.output.result.verdict' run.json
jq '.output.result.blockers' run.json
autoheal artifacts get "$(jq -r '.output.result.review_artifact_id' run.json)" --include content

Check that verdict is merge, merge_with_changes or do_not_merge, and that blockers is a list. With max_lines set to 10, verdict is not_reviewed and blockers is [].

Step 6: Read the fields from another agent​

An agent step runs another agent of the project, and its output is that agent's result. In the editor, click New → New agent and name it merge-gate. Replace its agent.yaml with this:

YAML
apiVersion: agents.autoheal.ai/v1
name: Merge gate
description: Runs pr-review-steps on one pull request and fails when the verdict is do_not_merge.
inputs:
prompt:
type: string
required: true
description: The pull request URL.
steps:
- id: review
agent: agents:pr-review-steps
with:
prompt: ${{ inputs.prompt }}
- id: gate
runtime: bash
env:
- name: VERDICT
value: ${{ steps.review.output.verdict }}
- name: BLOCKERS
value: ${{ steps.review.output.blockers }}
code: |
echo "Verdict: $VERDICT" >&2
printf '{"verdict": "%s"}' "$VERDICT" > "$AUTOHEAL_OUTPUT"
if [ "$VERDICT" = "do_not_merge" ]; then
echo "Blockers: $BLOCKERS" >&2
exit 1
fi

Commit, and run merge-gate on the example pull request. When the verdict is do_not_merge, the gate step fails, and the run fails. Check that the error of the gate step shows the verdict and the list of blockers. A list or an object in env arrives as JSON text.

A trigger starts an agent, but it does not read the agent's result. To act on the verdict of a review that a trigger starts, make the trigger start merge-gate instead of pr-review-steps.

What to know​

  • In returns, write required: true on each field. Do not write a JSON Schema required: [a, b] list. For an array, mark the array field, not its items. See Nested objects and arrays.
  • returns changes only the run's result. A follow-up message in the run's chat answers in text.
  • A step in then or else cannot be read by its own ID from outside the if step. ${{ steps.review.output.verdict }} in outputs passes the push, but the run fails with cannot fetch output from <nil>. Read steps.size.output.steps.review instead.
  • Without ?. and ??, an output that reads the branch that did not run fails the run, for example with cannot fetch verdict from <nil>.
  • When an agent has outputs, its result is the outputs object. Without outputs, its result is the output of its last step. On page 1, that was the if step's {branch, steps} object.
  • When a code step exits with a code that is not 0, the step's error is what the script wrote to stderr. The step has no output, even if it wrote $AUTOHEAL_OUTPUT.
  • agent: agents:pr-review-steps runs the agent in its own sandbox. It does not see the files of the step that calls it.

From the CLI​

BASH
autoheal agents push agents/pr-review-steps -m "Return a verdict and save the review as an artifact"
autoheal run pr-review-steps "https://github.com/autohealai/astronomy-shop-template/pull/1" --commit <sha> --wait -o json
autoheal agents push agents/merge-gate -m "Fail on a do_not_merge verdict"
autoheal run merge-gate "https://github.com/autohealai/astronomy-shop-template/pull/1" --commit <sha> --wait -o json

autoheal agents push pushes the whole agent folder, the artifact template included.

See Structured results, Step outputs and Calling agents and tools in the reference.