GitHub Actions

The auditor ships as a GitHub Action backed by a container on GHCR. It audits the repository on each pull request, writes findings onto the diff as inline annotations, and optionally pushes verified drift to Slack or Telegram.

You do not have to write any of this by hand. init reads the repository and writes the whole file, including the branches to audit and every optional input already wired to a secret: see the Quickstart. This page is the reference for what it wrote, and for changing it afterwards.

Minimum setup

No API key, no secret. The deterministic layer alone scores F1 0.889 against the project’s own evaluation, so this is worth running before you decide anything about model spend.

.github/workflows/contract-audit.yml

name: Contract audit
on: [pull_request]

permissions:
  contents: read
  security-events: write   # required for the SARIF upload

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7

      - id: audit
        uses: samso9th/contract-auditor@v1
        continue-on-error: true          # report first, block later
        with:
          spec: openapi.json
          source-dir: internal
          strip-prefix: /api/v1
          fail-on: none

      - uses: github/codeql-action/upload-sarif@v4
        if: always()
        with:
          sarif_file: ${{ steps.audit.outputs.sarif }}

Findings now appear as annotations on the exact lines of the pull request diff.

A Markdown summary is written to the run’s job summary automatically, so there is something readable even before the SARIF upload is configured.

Getting the three paths right

Most first runs fail on one of these.

Input	What it means	Finding yours
`spec`	The OpenAPI document your integrators build against	Commonly `openapi.json`, `docs/openapi.json`, `api/openapi.yaml`, or wherever your generator writes it
`source-dir`	Where route registrations and handlers live	Per language, below
`strip-prefix`	The prefix present in code but absent from spec paths	If code registers `/api/v1/payouts` and the spec documents `/payouts`, this is `/api/v1`

The last two are what differ between projects:

Your project	`source-dir`	`strip-prefix`	Detected by
Go (net/http, gin, chi, echo)	`internal`, `cmd`, or the module root	`/v1`	`go.mod`, or any `.go` file
TypeScript (Express)	`src`	`/api/v1`	`package.json`, or any `.ts` file
Python (FastAPI, Flask)	the package your app lives in, commonly `app`	`/api/v1`	`requirements.txt`, `pyproject.toml`, `setup.py`, `Pipfile`
PHP (Laravel)	the project root, so both `routes/api.php` and `app/Http/Controllers` are readable	`/api`	`artisan`, `composer.json`, `routes/api.php`

language is detected from the markers in the last column and only needs setting in a polyglot repository, where the first marker found wins and may not be the one you meant.

If a first run reports that every route is missing from the spec, strip-prefix is wrong. Your spec’s servers[].url usually names it.

Telling the contract apart from the dashboard

Most codebases register two APIs in one place: the one integrators hold an API key for, and the one a dashboard or admin console talks to with a session token. Only the first was ever promised to anyone. Their paths do not separate them, which is why a list of path globs goes stale the week after it is written. What separates them is which guard they sit behind.

      - uses: samso9th/contract-auditor@v1
        with:
          spec: docs/openapi.json
          source-dir: src
          strip-prefix: /api/v1
          contract-middleware: authenticate

Only routes registered with that middleware are audited, on both sides. An operation that the spec documents and a non-contract guard protects will not then read as missing from the code, and a route added next week lands on the right side without anyone editing a list. Both registration styles are read:

router.use(authenticate);                          // guards the whole router
router.get("/orders", listOrders);                 // in the contract
router.get("/stats", adminAuthenticate, getStats); // out of it

Route middleware is extracted for every supported language. Where a project supplies none, the run fails and says so instead of quietly excluding every route. Use exclude-paths there instead.

What the guard tells you beyond the filter

Restricting the audit to one guard also answers a question nobody asks in code review: what can this credential actually reach? The findings that come back are exactly the endpoints behind the guard that reads the credential your spec promises integrators. An endpoint you believed was dashboard-only appearing in that list is a finding about your architecture, not your documentation. It happens for an ordinary reason. A guard written to accept an API key or a session token protects every route behind it with both, so a key issued to move money also reaches account management. Nothing contradicts it, because nothing wrote it down. Two rules report the clear-cut cases directly, wherever route middleware is recorded:

Kind	Severity	Fires when
`auth_guard_missing`	critical	the spec declares security for the operation and the route carries no guard that reads any credential the spec names
`auth_guard_undocumented`	high	the route is guarded and the spec documents the operation as public, so an integrator following it is answered 401

Both are reported per file and only for files that guard something, so a project applying authentication once at application level gets silence rather than a page of false alarms.

router.use(mw) guards only the routes registered after it. A route above that line is open however protected the rest of the file looks, and that is the shape auth_guard_missing exists to catch.

Routes that are not part of the contract

Most codebases register endpoints no integrator was ever promised: a dashboard’s own session routes, internal health checks, an admin surface. Reporting them as drift is not wrong, just irrelevant, and a report that is mostly irrelevant stops being read.

      - uses: samso9th/contract-auditor@v1
        with:
          spec: docs/openapi.json
          source-dir: src
          strip-prefix: /api/v1
          exclude-paths: |
            /auth/*
            /internal/*

Patterns are matched against the path as your spec writes it, so after strip-prefix has been removed, and * crosses slashes: /auth/* covers /auth/me/password. A trailing /* covers the collection itself as well, so /auth/* also excludes /auth while leaving /authorize alone. Commas work as well as newlines, for a short list on one line. An excluded path leaves the audit in both directions. It counts as neither missing from the spec nor missing from the code, because the spec is not wrong to stay quiet about a route you have declared internal.

Excluding every endpoint fails the run rather than reporting a clean audit. The usual way to do that by accident is writing the prefix back in: /api/v1/* matches nothing, because the prefix is already gone by the time patterns are applied.

Adding the judgment pass

The deterministic rules settle everything mechanical. Three kinds of drift need reading comprehension instead: a handler that requires a field the spec calls optional, a default that quietly changed, and a validation bound loosened below what is documented.

      - id: audit
        uses: samso9th/contract-auditor@v1
        with:
          spec: docs/openapi.json
          source-dir: src
          language: typescript
          strip-prefix: /api/v1
          api-key: ${{ secrets.OPENROUTER_API_KEY }}
          workers: "16"
          fail-on: critical

Add OPENROUTER_API_KEY under Settings → Secrets and variables → Actions. Any OpenAI-compatible endpoint works; set base-url to point elsewhere. API keys and secrets covers where each credential comes from, and why a pull request from a fork never receives one.

Every claim, from the rules and the model alike, is executed against your handler before it reaches the report. In the project’s own evaluation the model’s raw precision was 0.23; the gate refuted every false claim and the report came out at precision 1.0. What you end up reading has been checked against your own code. See the verification gate.

Cost across the full 16-case evaluation was $0.044 for 159 model calls. One repository audit is a fraction of that.

Handing findings to a coding agent

Every run produces a fix brief: one document covering every finding, what was observed, and the test that proved it. There are three ways to use it, and none of them need storage set up. The files are uploaded as a GitHub Actions artifact, which needs no bucket, no account and no credentials.

      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: contract-audit-brief
          path: ${{ steps.audit.outputs.brief-dir }}

Copy it
Download the brief
Download the zip

The pull request comment contains the whole brief in a collapsible block. Expand it, copy it, paste it into Cursor, Codex or Claude Code. Nothing to download.Best when you want a fix started immediately and do not care about running the tests yourself.

<name>_brief.md from the run’s Artifacts section. The same document as a file, for attaching to a ticket or keeping alongside a branch.

<name>_brief.zip contains the brief and a tests/ directory holding every generated test as a runnable file.This is the most useful option. You do not have to tell the agent what “fixed” means, because each test already asserts what the specification promises and currently fails. When it passes, that finding is resolved.

The brief opens by telling the agent the one judgement the tool deliberately leaves alone: whether the code drifted or the document went stale. Getting that backwards turns a documentation edit into a breaking change for everyone already integrated, so the brief asks the agent to decide explicitly and say why.

Inputs

Input	Default	Notes
`spec`	n/a	Required. Path to your OpenAPI document.
`source-dir`	`.`	Directory containing the API source.
`language`	`auto`	`go`, `typescript`, or detect.
`strip-prefix`	n/a	Prefix to strip so routes match spec paths.
`contract-middleware`	n/a	Only audit routes guarded by these middleware, one name per line. Also enables the two guard rules.
`exclude-paths`	n/a	Globs to leave out of the audit, one per line. Matched after `strip-prefix` is removed.
`api-key`	n/a	Omit for deterministic-only: no cost, no secret.
`base-url`	OpenRouter	Any OpenAI-compatible endpoint.
`model`	`z-ai/glm-5.3-flash`	Any model id your endpoint serves.
`reasoning`	n/a	`off`, `low`, `medium`, `high`. Slower and dearer; omit for the model’s own default.
`workers`	`8`	Concurrent model calls. Raise to shorten wall clock.
`fail-on`	`high`	`critical`, `high`, `medium`, `low`, `none`.
`memory-url`	n/a	Turns on self-improvement. Storage you own; nothing is kept in your repo.
`memory-key-id`	n/a	Access key id, or Cloudinary API key. Pass a secret.
`memory-secret`	n/a	Secret key, or Cloudinary API secret.
`memory-token`	n/a	Bearer token for an HTTP store, or an IPFS pinning JWT.
`memory-region`	n/a	Region for an S3-compatible store.
`webhook-url`	n/a	Full JSON report POSTed to a data sink.
`webhook-secret`	n/a	Sent as `X-Auditor-Token`.
`slack-webhook-url`	n/a	Formatted Slack message, verified findings only.
`telegram-bot-token`	n/a	With `telegram-chat-id`.
`notify-min-severity`	`high`	Lowest severity worth interrupting a human for.

Outputs: findings, critical, high, sarif, summary, brief, brief-zip, brief-dir.

Turning on self-improvement

The auditor can learn from its own mistakes across runs, but only if you tell it where to keep the record. There is no default location and no shared store. Leave memory-url out and it keeps no memory at all.

      - uses: samso9th/contract-auditor@v1
        with:
          spec: openapi.json
          api-key: ${{ secrets.OPENROUTER_API_KEY }}
          memory-url: s3://my-bucket/contract-auditor
          memory-key-id: ${{ secrets.MEMORY_KEY_ID }}
          memory-secret: ${{ secrets.MEMORY_SECRET }}
          memory-region: eu-west-1

Any S3-compatible bucket works. For Cloudflare R2, MinIO, Spaces or Backblaze, add the endpoint to the URL:

          memory-url: s3://my-bucket/contract-auditor?endpoint=https://<account>.r2.cloudflarestorage.com

Other backends, same shape:

Store	`memory-url`	Credentials
Any HTTPS endpoint that takes GET and PUT	`https://host/ledger.jsonl`	`memory-token`
Cloudinary raw storage	`cloudinary://cloud-name/ledger`	`memory-key-id`, `memory-secret`
IPFS via a Pinata-compatible pinner	`ipfs://ledger`	`memory-token`

Two things change when you set this. Each run reads the ledger before auditing and appends to it afterwards, so the store has to be reachable from the runner. And the verification gate starts running against your repository, because a claim with no verdict teaches nothing: a temporary test is written next to your code, executed, and deleted. A claim its test disproves is dropped from the report. One whose test cannot be built is kept and marked, never silently lost.

Memory only ever adjusts priors. It changes which past mistakes the model is shown and how findings are ranked, and it can never suppress a finding: if the generated test still fails, the finding ships. About one endpoint in twenty is audited with memory disabled on purpose, so a wrong assumption keeps getting tested instead of settling in permanently. Nothing is written into your checkout, and no ledger ships inside the action’s image: the build deletes any that reaches it and fails if one remains.

Alerting a human

Two different jobs, deliberately separate.

Slack or Telegram
Data sink
PR comment

Formatted, verified-only, and silent on a clean run.

    with:
      slack-webhook-url: ${{ secrets.SLACK_WEBHOOK_URL }}
      notify-min-severity: critical

See notifications for the formatting and filtering rules.

Raw JSON, every finding, for Postgres or a warehouse.

    with:
      webhook-url: ${{ secrets.AUDITOR_WEBHOOK_URL }}
      webhook-secret: ${{ secrets.AUDITOR_WEBHOOK_SECRET }}

file and line are structured fields, so a sink never parses prose to place a finding.

{
  "tool": "contract-auditor",
  "repository": "owner/repo",
  "sha": "abc123",
  "run_url": "https://github.com/owner/repo/actions/runs/123",
  "summary": { "total": 3, "by_severity": { "critical": 1, "high": 2 } },
  "findings": [
    {
      "path": "/payouts", "method": "post",
      "kind": "response_type_mismatch", "detail": "amount",
      "severity": "critical",
      "file": "handlers/types.go", "line": 27,
      "evidence": "PayoutResponse.Amount is float64 (JSON number); spec declares string",
      "verdict": "confirmed"
    }
  ]
}

    - uses: marocchino/sticky-pull-request-comment@v2
      if: github.event_name == 'pull_request'
      with:
        path: ${{ steps.audit.outputs.summary }}

Needs pull-requests: write in the job’s permissions.

Rolling it out to a team

Report only

fail-on: none with continue-on-error: true. Nothing blocks.

Put findings on the diff

Add the SARIF upload, so they appear where people already look.

Clear the backlog

Fix what is real. Record deliberate differences in auditor/memory/allowlist.json. Each entry needs a reason and a date, so the allowlist does not become a place where findings go to be forgotten.

Tighten

fail-on: critical, then high once the signal is trusted.

Do not block merges on day one. That is the quickest way to get the tool removed in week two.

Scheduled audits

Pull request runs catch drift as it is introduced. A weekly run catches the drift that was already there before you adopted the tool, and drift in the spec itself.

on:
  schedule:
    - cron: "0 9 * * 1"    # Mondays, 09:00 UTC
  pull_request:

Troubleshooting

Every route reported as missing from the spec

strip-prefix is wrong. If code registers /api/v1/payouts and the spec documents /payouts, set strip-prefix: /api/v1.

The report is mostly routes that were never meant to be public

Dashboard session routes, health checks and admin surfaces are registered in code and deliberately absent from an integrator-facing spec, so every audit reports them. Leave them out with exclude-paths, one glob per line, matched against the path as the spec writes it.

manifest unknown when pulling the image

The package is private, or was never linked to its repository. Check with gh api /user/packages/container/contract-auditor --jq '.visibility'. Publishing from a public repository normally yields a public package automatically; if it reports private, set Package settings → visibility → Public.

denied: permission_denied when publishing

Settings → Actions → General → Workflow permissions → Read and write. The workflow requesting packages: write is not enough on its own; the repository default caps what the token can be granted.

Findings show verdict: error

The gate could not compile or run its generated test, usually a Go toolchain mismatch: the image pins Go 1.24. These findings are kept and flagged, never silently dropped, because a broken toolchain must never look like a clean bill of health. Deterministic findings are unaffected; they execute nothing.

The run reported unread_endpoints

The model returned something unparseable for those endpoints even after a retry, so the judgment pass did not read them. That gets reported rather than hidden, because an endpoint nobody read is not the same as an endpoint with no drift.

TypeScript findings show unsupported

Route extraction works for TypeScript; the verification gate does not yet. Unverified claims are not let through, so they are reported as unsupported.

Getting started

How it works

Reference

Minimum setup

Getting the three paths right

Telling the contract apart from the dashboard

What the guard tells you beyond the filter

Routes that are not part of the contract

Adding the judgment pass

Handing findings to a coding agent

Inputs

Turning on self-improvement

Alerting a human

Rolling it out to a team

Scheduled audits

Troubleshooting

​Minimum setup

​Getting the three paths right

​Telling the contract apart from the dashboard

​What the guard tells you beyond the filter

​Routes that are not part of the contract

​Adding the judgment pass

​Handing findings to a coding agent

​Inputs

​Turning on self-improvement

​Alerting a human

​Rolling it out to a team

​Scheduled audits

​Troubleshooting

Minimum setup

Getting the three paths right

Telling the contract apart from the dashboard

What the guard tells you beyond the filter

Routes that are not part of the contract

Adding the judgment pass

Handing findings to a coding agent

Inputs

Turning on self-improvement

Alerting a human

Rolling it out to a team

Scheduled audits

Troubleshooting