Facts
Claims about observed behavior—exit status, streams, hangs, flags, bytes, and schemas—and a readiness hypothesis.
Boundary: It never licenses “an agent passed” and is never used to rank the leaderboard.
published scoring contract · frozen before public scores
Rubric v1.0.0
Nine conceptual dimensions. Seven are scored in the public cold lane; D5 input fuzzing and D8 shipped agent surfaces remain N/A until package introspection can test them fairly.
7 cold-scored dimensions72 principles mapped2 score types, never conflated0 points lost to N/A
score surface
Seven public dimensions
| dimension | name | max | question |
|---|---|---|---|
| D1 | Legibility | 15 | can an agent learn this tool from its own surface? |
| D2 | Structured I/O | 20 | is the output machine-parseable? |
| D3 | Non-blocking | 20 | does it ever hang when no human is present? |
| D4 | Context economy | 15 | how many tokens does it cost to use? |
| D6 | Safety rails | 10 | can an agent preview before it destroys? |
| D7 | Error recovery | 20 | does a mistake produce an honest, actionable signal? |
| D9 | Cold credentials | 20 | can an agent get authenticated without a human? |
Grades are deterministic summaries only: A ≥ 90 · B ≥ 80 · C ≥ 70 · D ≥ 60 · F ≥ 0. They do not rank the leaderboard.
claim discipline
Two score types and what each licenses
Verdict
Claims that an agent completed or failed the sampled task, with median cost and cross-trial variance.
Boundary: It does not claim universal success across every task, version, model, or credential state.
transparent and fair beats gotcha
Fairness rules
- 01
N/A never penalizes. A non-applicable check is excluded from both numerator and denominator.
- 02
Subcommand help is unioned with top-level help because many CLIs document JSON, confirmation, and field flags only where they apply.
- 03
Filters, REPLs, and other tools designed to read stdin are not penalized for waiting on a non-delivering pipe.
- 04
Read-only tools are not penalized for missing dry-run or confirmation flags.
- 05
Every verdict requires at least three isolated trials with the same prompt and links to the raw trajectory.
source coverage · 72 of 72
The principle map
Ranges below partition all 72 principles exactly once. “Auth” checks are N/A in a cold run, not failures.
| principles | dimension | mapping | coverage |
|---|---|---|---|
| 1–3 | Frame | Human DX and agent DX serve different operators; support both paths. | advisory |
| 4–12 | D2 | Structured request payloads, JSON output, and streamable non-TTY defaults. | cold |
| 13–18 | D1 | Runtime schema, self-description, types, scopes, and canonical capability discovery. | cold |
| 19–21 | D4 | Field masks and response shaping conserve agent context. | cold |
| 22–23 | D2 | NDJSON pagination and incremental processing. | cold / auth |
| 24–25 | D4 | Shipped guidance teaches context conservation. | static |
| 26–38 | D5 | Treat agent input as adversarial: traversal, controls, query fragments, and double encoding. | cold / advisory |
| 39–40 | D8 | Ship structured skill files for each surface and workflow. | static |
| 41 | D1 | Agent guidance must be reachable from the tool itself. | static |
| 42–50 | D8 | Encode invariants and expose schema-derived MCP and native agent surfaces. | cold / static |
| 51–55 | D3 / D9 | Credential injection and headless authentication must not require a browser. | cold / static |
| 56–61 | D6 | Dry-run, confirmation, sanitization, and response prompt-injection defenses. | cold / auth |
| 62 | D2 | Retrofit machine-readable JSON output first. | cold |
| 63 | D5 | Then validate agent-supplied input defensively. | cold |
| 64 | D1 | Then expose a queryable schema. | cold |
| 65 | D4 | Then add field selection and context controls. | cold |
| 66 | D6 | Then add dry-run for mutation. | cold |
| 67–68 | D8 | Then ship skills and an MCP surface from the same source. | cold / static |
| 69–70 | D5 | Fuzz agent-typical mistakes, double encoding, and control characters. | cold |
| 71–72 | D6 | Verify dry-run validation and defend against prompt injection in retrieved data. | cold / auth |
versioning policy
Scores never silently change meaning
Clarifies wording or fixes an implementation bug without changing intended scoring. Existing reports stay pinned.
Adds or reweights a check. A new rubric version is published first and affected corpus reports are rerun.
Changes a dimension or what a score licenses. Old permalinks remain readable with their original rubric version.