> ## Documentation Index > Fetch the complete documentation index at: https://contract-auditor.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Command reference > Every entry point, and what it costs. ## Make targets | Target | Does | Key | Cost | | -------------------- | ------------------------------------------------------------------ | ----- | ----------------------- | | `make cases` | Build the 16 evaluation cases | no | none | | `make test-tools` | Test every deterministic component | no | none | | `make check` | Confirm the fixture builds and the scorer is sound | no | none | | `make routes` | Print the route table for the clean fixture | no | none | | `make deterministic` | Run the no-model layer and score it | no | none | | `make verify` | Put every deterministic finding through the gate | no | none | | `make models` | Show configured models, prices and how to pick another | no | none | | `make test-llm` | Check the model client against the live endpoint | yes | `<$0.001` | | `make baseline` | Run the single-prompt baseline | yes | see [results](/results) | | `make agent` | Run the full auditor | yes | see [results](/results) | | `make score` | Score both runs side by side | no | none | | `make memory` | Show how often each claim kind has survived the gate | no | none | | `make memory-check` | Prove the configured memory store round-trips | store | none | | `make rules` | Promote repeated dismissals into rules, and re-check existing ones | no | none | | `make harvest` | Grow the evaluation set from the last run | no | none | | `make self-improve` | Run the cases twice and compare the two runs | yes | \~`$0.15` | | `make notify` | Preview the alert for the latest run | no | none | | `make clean` | Remove generated cases and runs | no | none | ## Direct commands Generate the workflow by reading a repository, instead of writing it by hand: ```bash theme={null} python3 auditor/init.py --repo /path/to/api ``` It writes `.github/workflows/contract-audit.yml` with every derived value commented, refuses to overwrite an existing one without `--force`, and takes `--stdout` to print instead. `--json` reports what it detected without writing anything, which is the quickest way to see whether it found your spec. Audit a real repository: ```bash theme={null} python3 auditor/run.py --repo /path/to/go/api --spec /path/to/openapi.json \ --docs /path/to/docs --out reports/runs/my-api ``` Leave out routes that were never part of the contract. Patterns are matched against the path as the spec writes it, so after `--strip-prefix` has been removed, and `*` crosses slashes: ```bash theme={null} python3 auditor/run.py --repo /path/to/api --spec openapi.json \ --strip-prefix /api/v1 --exclude-paths '/auth/*,/internal/*' \ --out reports/runs/my-api ``` The same flag works on the deterministic layer on its own: ```bash theme={null} python3 auditor/tools/diff.py /path/to/api openapi.json \ --strip-prefix /api/v1 --exclude-paths '/auth/*' --json ``` Extract the route table: ```bash theme={null} python3 auditor/tools/routes.py /path/to/go/api --strip-prefix /v1 ``` Verify one case: ```bash theme={null} python3 auditor/verify.py eval/cases/D01 ``` Score a run: ```bash theme={null} cd eval && python3 score.py --run ../reports/runs/agent --markdown ``` Compare two runs, which is how the learned memory is measured: ```bash theme={null} python3 eval/compare.py reports/runs/memory-1 reports/runs/memory-2 ``` ## Choosing a model Any model your endpoint serves works: the client sends whatever id it is given, and nothing in the auditor is tied to one vendor. `--model` sets it per run, `AUDITOR_MODEL` in `.env` sets it once. `--reasoning` asks the provider to think for longer before answering. It costs output tokens and wall-clock time on every endpoint, so it is off unless you ask for it, and the published [results](/results) were measured without it. ```bash theme={null} python3 auditor/llm.py --models # what is configured, and prices python3 auditor/run.py --model anthropic/claude-sonnet-5 --repo . --spec openapi.json python3 auditor/run.py --reasoning high --cases eval/cases --out reports/runs/deep ``` ```bash theme={null} # .env, to set both once AUDITOR_MODEL=deepseek/deepseek-v3.2 AUDITOR_REASONING=medium ``` | Level | Sends | Use when | | ------------------------- | ----------------------------- | ------------------------------------------------------- | | omitted | no reasoning field at all | Default. The model does whatever it normally does | | `off` | `reasoning: {enabled: false}` | You want speed and the model lets you turn thinking off | | `low` / `medium` / `high` | `reasoning: {effort: ...}` | Judgment calls are being missed and you will pay for it | Not every model accepts every level. `z-ai/glm-5.3-flash` reasons unconditionally and rejects `off` with a 400; the error names the model and the level so it is clear what to change. Asking for a level also raises the token ceiling, because thinking is billed as output and emitted before the answer, so a ceiling sized for the answer alone truncates mid-thought. Both settings are recorded in every run's `meta`, since two runs at different models or reasoning levels are not comparable. ## Learned memory Off unless you point it at storage you own. Nothing is kept in your repository or inside the action's image, and no history ships with the tool. ```bash theme={null} export AUDITOR_MEMORY_URL=s3://my-bucket/contract-auditor export AUDITOR_MEMORY_KEY_ID=... # or AWS_ACCESS_KEY_ID export AUDITOR_MEMORY_SECRET=... # or AWS_SECRET_ACCESS_KEY python3 auditor/memory/store.py --check # write a probe, read it back ``` | URL | Backend | | ------------------------------ | ------------------------------------------------------------------------------------------------------ | | `s3://bucket/prefix` | Any S3-compatible store. Add `?endpoint=https://...` for R2, MinIO, Spaces, B2, and `?region=` for AWS | | `https://host/ledger.jsonl` | Any endpoint answering GET and PUT. `AUDITOR_MEMORY_TOKEN` becomes a bearer token | | `cloudinary://cloud/public_id` | Cloudinary raw storage, with `CLOUDINARY_API_KEY` and `CLOUDINARY_API_SECRET` | | `ipfs://name` | IPFS through a Pinata-compatible pinning API, with `AUDITOR_MEMORY_TOKEN` as the JWT | | `file:///abs/path` | A local file. Development only: a ledger in a working tree is one `git add` from being published | Every run appends each claim and its gate verdict, refuted claims included. Later runs read it back: similar past refutations go into the prompt, and per-kind survival rates decide the order of the report. Memory never suppresses a claim; the gate still decides. ```bash theme={null} python3 auditor/memory/ledger.py --verdict refuted # the labelled false positives python3 auditor/memory/recall.py # calibration per claim kind python3 auditor/memory/recall.py --endpoint "post /payouts" # what would be recalled python3 auditor/memory/rules.py # shapes dismissed 3+ times python3 eval/harvest.py --run reports/runs/agent # new cases from real findings ``` | Flag on `auditor/run.py` | Does | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `--memory-url URL` | Where the ledger lives. Defaults to `AUDITOR_MEMORY_URL`; unset means no memory at all | | `--no-memory` | Ignore learned history for this run. The ledger is still written | | `--epsilon 0.05` | Fraction of endpoints audited with memory off, so a wrong prior keeps being tested. `0` disables it | | `--verify` | With `--repo`: put every claim through the gate. Implied when memory is configured, because a claim with no verdict teaches nothing | | `--exclude-paths GLOBS` | Paths to leave out of the audit, newline or comma separated. Also on `auditor/tools/diff.py` | | `--contract-middleware NAMES` | Only audit routes guarded by these middleware. Also on `auditor/tools/diff.py` | ## Configuration From `.env` at the repository root, or the environment, which wins. ```bash theme={null} OPENROUTER_API_KEY=sk-or-... OPENROUTER_BASE_URL=https://openrouter.ai/api/v1 SLACK_WEBHOOK_URL=https://hooks.slack.com/services/... TELEGRAM_BOT_TOKEN=123456:ABC... TELEGRAM_CHAT_ID=-1001234567890 ```