Ultrareview in CI

This page is one working recipe and the three ways it goes wrong. It assumes you have read Ultrareview from the terminal, which covers the commands themselves.

Do you need a CI job at all?

If ultrareview already reviews your pull requests, you get the review either way: it is posted to the pull request from our side, with no runner involved. A CI job buys two things that cannot be had otherwise.

• A check the build reads. A step that exits non-zero is a red check, and a red check can be required.
• Code-scanning alerts. Findings become native GitHub alerts, with the annotations and the security tab that come with them.

If you want neither, you do not need this page.

The shape of it

Three steps, and the split matters:

1. Ask for the review, which returns a run id immediately.
2. Wait for the run to end. This is its own step because the reading step has no wait of its own.
3. Read the findings once, deciding both the exit code and the SARIF file.

The reason the last two are not one step is worth stating plainly: pr findings has no --wait. It reads a verdict, it does not produce one, and asked about a review that is still running it exits 3 rather than reporting a part-written list as a result. --wait is a flag the parser knows, so passing it to pr findings is accepted and does nothing at all, which is the quietest possible way to get this wrong.

The workflow

name: Ultrareview
on: pull_request

permissions:
  contents: read
  # Required by upload-sarif. Without it the upload step fails with a 403.
  security-events: write

jobs:
  review:
    # A pull request from a FORK, and a Dependabot one, get no repository secrets. Without this the token
    # is empty, the CLI exits 4 (not authenticated), and every such pull request goes red for a reason
    # that has nothing to do with the code in it. Skipping is the honest outcome: we cannot review what we
    # cannot authenticate for, and a skipped job says so where a failed one misleads.
    if: >-
      github.event.pull_request.head.repo.full_name == github.repository &&
      github.actor != 'dependabot[bot]'
    runs-on: ubuntu-latest
    env:
      ULTRAREVIEW_TOKEN: ${{ secrets.ULTRAREVIEW_TOKEN }}
      REPO: ${{ github.repository }}
      PR: ${{ github.event.pull_request.number }}
    steps:
      - run: npm i -g ultrareview

      - name: Ask for the review
        id: start
        run: |
          RUN=$(ultrareview pr review --repo "$REPO" --pr "$PR" --json | jq -r .runId)
          # jq exits 0 on a refusal, so the pipe swallows a failed request. This is the check that does not.
          test -n "$RUN" && test "$RUN" != "null"
          echo "run=$RUN" >> "$GITHUB_OUTPUT"

      - name: Wait for it to finish
        # The gate is the next step, which re-reads the run and decides honestly. This step only blocks
        # until there is something to read, so a timeout or a failed run must not stop the job here.
        run: ultrareview pr status --run "${{ steps.start.outputs.run }}" --wait --timeout 20m || true

      - name: Read the findings
        id: findings
        run: |
          code=0
          ultrareview pr findings \
            --run "${{ steps.start.outputs.run }}" \
            --fail-on major \
            --format sarif > ultrareview.sarif || code=$?
          echo "code=$code" >> "$GITHUB_OUTPUT"

      - name: Upload to code scanning
        # 0 and 1 are the only two codes that mean a review of this commit was read end to end, and
        # therefore the only two that write SARIF. See the guard below.
        if: steps.findings.outputs.code == '0' || steps.findings.outputs.code == '1'
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: ultrareview.sarif

      - name: Fail on findings, or on a review that could not certify
        run: exit ${{ steps.findings.outputs.code }}

Why the job skips fork and Dependabot pull requests

GitHub does not give repository secrets to a workflow run from a fork, or to Dependabot, and it is right not to: a pull request from a stranger could otherwise read your token. So ULTRAREVIEW_TOKEN arrives empty, the CLI exits 4 because it cannot authenticate, and the pull request goes red for a reason that has nothing to do with the code in it.

Skipping says the true thing. If you want fork pull requests reviewed, trigger the review from a workflow that runs on your side of the fence (pull_request_target, or a scheduled sweep) and understand what you are granting before you do: pull_request_target runs with your secrets against a stranger's branch.

Why the upload is guarded

An empty SARIF run is not nothing. It is a positive statement that this tool looked and found nothing, and GitHub acts on it: uploading an empty run CLOSES every alert that tool previously raised. That is exactly what you want after a clean review, and it is the reason a clean review emits an empty log rather than no output at all. Upload nothing and every fixed alert stays open forever.

Now put that together with the obvious-looking way to write the upload step:

# Do not do this.
- uses: github/codeql-action/upload-sarif@v3
  if: always()
  with:
    sarif_file: ultrareview.sarif

if: always() runs the upload when the review failed, when it was skipped, when it lost coverage, and when the runner could not reach the service. Every one of those has zero findings, and none of them means the code is clean. Uploaded as an empty SARIF run, each would close every open ultrareview alert in the repository, silently, because a network blip.

The CLI will not let that happen from its side: it writes SARIF only for exit 0 and exit 1, and writes a document that is deliberately not SARIF for everything else, so an unguarded upload step fails loudly rather than succeeding destructively. Guarding on the exit code is the same rule stated on your side, one step earlier, where it produces a skipped step instead of a red one.

Read the exit code, never the pipe

# Wrong: jq exits 0, so the review's verdict is thrown away.
ultrareview pr findings --run "$RUN" | jq .findings

# Right: the code is the answer, so capture it before anything else touches it.
ultrareview pr findings --run "$RUN" --json > findings.json || code=$?
jq .findings < findings.json

A shell pipeline reports the exit status of its last command. Pipe the gating command into anything, jq or tee or head, and the build reads the exit code of that instead. This is not a hypothetical: it is the single easiest way to build a gate that always passes.

The same applies to set -e. A step running under bash -e, which is what GitHub Actions gives you, dies at the first non-zero exit, so cmd || code=$? is how you both survive the failure and keep its code.

Which codes should stop a build

The one that surprises people is 3. The field convention is to treat "nothing to report" as success, and this tool does the opposite: a review that never ran, never finished, or covered less than it should is a red build, because a green check on a review that never read the code is worse than a red one. If you want a softer posture, take it one case at a time (--allow-stale reads a review of a different commit) rather than by treating 3 as a pass.

Retrying a 5 is worth doing and retrying anything else is not: a rejected credential, a missing run and a forbidden scope are all answers, and asking again produces the same answer more slowly.

The token

Mint one in the dashboard, under Configuration and then API tokens, and hold it as a repository or organization secret. It needs both scopes: reviews:run to start the review and reviews:read to read what it found. Scopes are fixed when a token is minted, so a token missing one is replaced, never upgraded.

Two things to set up front rather than discover:

• Limit it to the repositories that use it. A repository outside the limit answers exactly like one that does not exist, so the build sees exit 2 and a "no such repository" message rather than a permission error.
• Watch the expiry. Tokens expire (90 days by default) and a build that has worked for three months starts failing with exit 4 on a Tuesday morning for no reason anybody remembers.

ultrareview doctor is the diagnostic for both, and it is worth running as its own step in a nightly job rather than only when something breaks. It fails while a token still works, seven days ahead of expiry, which is the point: an expiry discovered as a failed call halfway through a job is too late to be a warning. It wants both CLI scopes, so a deliberately read-only token will fail its scope check with exit 7; read the named scope rather than the code if that is the setup you meant to have.

Waiting, and what it costs

--wait polls every 5 seconds and gives up after 15 minutes unless you say otherwise. Both are worth setting explicitly in CI:

ultrareview pr status --run "$RUN" --wait --timeout 20m --poll 10s

A timeout is a fact about the wait and never about the review: the run carries on, and reading it later tells you how it ended. A very short --poll is not free either, since each poll is a request against the token's own rate limit, and several concurrent jobs sharing one token share that limit. A zero interval is refused rather than quietly floored.

Somewhere other than GitHub Actions

Nothing here is specific to Actions except the SARIF upload and the $GITHUB_OUTPUT plumbing. The shape is the same anywhere:

set -euo pipefail
export ULTRAREVIEW_TOKEN="$CI_ULTRAREVIEW_TOKEN"

# pipefail is what makes the pipe below safe: without it, jq's zero hides a failed review request.
RUN=$(ultrareview pr review --repo "$REPO" --pr "$PR" --json | jq -r .runId)
ultrareview pr status --run "$RUN" --wait --timeout 20m || true

code=0
ultrareview pr findings --run "$RUN" --fail-on major --json > findings.json || code=$?
cat findings.json
exit "$code"

For a self-hosted installation, point it at your own host with ULTRAREVIEW_BASE_URL; everything else is identical.

Related

• Ultrareview from the terminal: every command and flag, and the full exit-code contract.
• API tokens: scopes, repository limits, and rate limits.