Methodology · workflow-benchmark@2
A result must describe
an observed outcome.
We evaluate representative workflows for 11 developer CLIs. This is a declared task suite, not a test of every command or a prediction of every agent deployment.
Tasks and trials
Each CLI has a normal scenario, an ambiguous or larger-state scenario, and a controlled-recovery scenario. Each runs three times with independent local and remote state. The planned campaign contains 99 primary journeys.
The planted fault belongs to Controlled recovery only. Normal and ambiguous journeys start from a healthy fixture so they remain usable baselines.
The catalog is grouped by product job — Deploy, Auth, Payments, Databases, Observability, File storage, Source control — not by a generic "service configuration" bucket. Auth0 is Auth; Stripe is Payments; Vercel, Wrangler, and Firebase are Deploy. Each scenario shows its own n/3 outcome. No result is averaged with interface readiness. Three repetitions measure this task only; they do not establish general reliability.
Agent environment
The agent can use shell and file tools in a disposable environment. Target-service operations must pass through the evaluated CLI. A trusted verifier independently inspects state, outside the agent's writable workspace.
Reference model: anthropic/claude-sonnet-4.6. Campaign contract: workflow-campaign@1. Limits: 40 model turns for the whole journey, 500,000 cumulative tokens across repeated context, 15 minutes, bounded commands, temperature 0.6. Spend is capped separately at $20. Seeds are recorded only where supported. Package versions, integrity, runtime, and harness revisions are attached to every result.
Verification and controls
Success requires the intended state on the correct target, preserved unrelated state, and no forbidden side effects. A reference command proves feasibility; alternative valid command sequences are accepted.
Required controls: reference, no-op, printed-success, wrong-resource, partial, tampered-artifact. Controls and agent trials use different disposable state. Controls are not counted as agent successes.
Incomplete evidence is not failure
We distinguish verified pass, verified task failure, not run, reviewed not applicable, blocked prerequisites, harness error, and provider error. An unattempted downstream stage is Not reached. A missing exit code remains unknown.
Infrastructure-invalid attempts are retained. Valid failures are never rerun selectively to improve a result.
Interface readiness · rubric 4.0.0
Mean of applicable domain scores; each domain is the fraction of applicable behavioral checks passed. All applicable checks require evidence before an overall percentage is published. No workflow outcome is included.
- Discovery and documentation
- Output usability
- Headless execution
- State changes, preview, and replay
- Errors and recovery
- Authentication and configuration
Help text can establish documented capabilities. It cannot establish replay safety, error recovery, preview safety, or authentication behavior. Optional MCP, skill files, and vocabulary choices earn no behavioral credit.
Weights are an explicit diagnostic policy, not a validated predictor. Incomplete measurement has no overall percentage, letter grade, or CLI rank. Historical scores retain their original methodology.
helpThe CLI documents how to start the assigned workflow.
Applicability: Always, when a help or equivalent discovery surface exists.
Exercise: Invoke documented discovery without changing fixture state.
Expected: Usable command names and required flags for the task.
Pass: An observation shows the agent can locate the relevant entrypoint from CLI output.
Measurement error: Timeout, refusal, or null exit stays unknown.
discoveryPrerequisites and configuration needed for the task are discoverable.
Applicability: When the workflow needs local or service configuration.
Exercise: Read documented config/help for the fixture before mutating.
Expected: The required config keys or files are named.
Pass: Observation shows those prerequisites without a successful mutation.
Measurement error: Missing help is fail only when the surface was invoked.
structured-outputRelevant machine output can be parsed in full.
Applicability: When the task emits list or object output.
Exercise: Capture the complete relevant stream, including every NDJSON record.
Expected: Parsed records match planted objects.
Pass: Every planted record is present; truncated streams are unknown.
Measurement error: Truncation is not a parse success.
clean-streamsStdout used for data is not mixed with interactive noise that blocks parsing.
Applicability: When structured or list output is in scope.
Exercise: Run the read/list path headlessly.
Expected: The verifier can parse the relevant stream.
Pass: Parse succeeds without stripping banners by hand.
Measurement error: Pager hangs are fail for headless, not unknown.
paginationA planted dataset larger than one page is retrieved completely.
Applicability: When the CLI lists resources and the fixture plants more than one page.
Exercise: Page through the planted set and union results.
Expected: All planted identifiers are recovered.
Pass: Completeness check against the planted set.
Measurement error: Documented pagination flags without traversal stay unmeasured.
help-completionDiscovery commands complete without a TTY.
Applicability: Always for required discovery.
Exercise: Run help/version/config listing under the harness.
Expected: Non-zero-or-zero exit is recorded; no hang.
Pass: Command completes with a recorded exit.
Measurement error: Null exit is unknown.
task-completionThe workflow command path can complete headlessly once configured.
Applicability: When a non-interactive path exists.
Exercise: Run the assigned mutate/verify path with the fixture.
Expected: No prompt hang; exit recorded.
Pass: The command sequence completes; success still depends on oracles.
Measurement error: OOM and timeout are harness/blocked, not silent pass.
read-safetyRead/status operations do not change relevant state.
Applicability: When a read/status path is in the workflow.
Exercise: Compare relevant state before and after the read stage.
Expected: Target and unrelated snapshots match.
Pass: Digest equality on the declared state.
Measurement error: Unobserved state is unknown, not pass.
previewPreview shows the intended change and leaves target state unchanged.
Applicability: When the CLI offers preview/diff/dry-run for the mutation.
Exercise: Preview, then re-observe state before applying.
Expected: Predicted change is inspectable; target unchanged.
Pass: Preview output is retained and state digests match.
Measurement error: Flag presence without a preview observation is unmeasured.
replayRepeating the successful mutation does not duplicate or damage state.
Applicability: When the workflow has a replay stage.
Exercise: Run the mutation twice from independent observations.
Expected: Intended state remains; no extra copies of the planted object.
Pass: Second-run oracles pass.
Measurement error: A documented read verb is not replay evidence.
invalid-commandAn invalid command is rejected honestly.
Applicability: When the CLI has a command parser.
Exercise: Invoke a known-invalid command.
Expected: Non-zero exit and an error on stderr or stdout.
Pass: Induced error observed.
Measurement error: No induced error means unmeasured, never pass.
invalid-flagAn invalid flag is rejected honestly.
Applicability: When flags are documented.
Exercise: Invoke a known-invalid flag on a real command.
Expected: Non-zero exit and an error message.
Pass: Induced error observed.
Measurement error: Success-path trials do not establish this check.
recoveryA known injected fault can be repaired to the intended state.
Applicability: When the recovery variant is in scope.
Exercise: Inject the declared fault, then require restoration.
Expected: Fault artifact gone; intended state present; unrelated preserved.
Pass: Recovery oracles pass.
Measurement error: Unattempted recovery is not-reached, not fail-invented.
missing-authMissing credentials fail closed.
Applicability: When the CLI supports authenticated service calls.
Exercise: Clear credentials and invoke a service operation.
Expected: Non-zero exit; no successful mutation.
Pass: Denied observation with unchanged state.
Measurement error: Local-only CLIs may be reviewed not-applicable.
invalid-authInvalid credentials fail closed.
Applicability: When credentials are used.
Exercise: Supply an invalid token.
Expected: Non-zero exit; no mutation.
Pass: Denied observation with unchanged state.
Measurement error: Not run stays unknown.
valid-authValid scoped credentials permit the assigned operation.
Applicability: When the workflow is authenticated.
Exercise: Use the evaluation credential.
Expected: The assigned operation can proceed.
Pass: Observation of a permitted call, not merely a login banner.
Measurement error: Login without a task operation is incomplete.
scope-authInsufficient scope is denied without widening access.
Applicability: When the vendor supports scoped tokens.
Exercise: Use a token missing the required permission.
Expected: Denied; unrelated resources unchanged.
Pass: Scoped denial observed.
Measurement error: Vendors without scope controls need a reviewed not-applicable rationale.
What is measured on launch
Public /api/scan stays credential-free with deny-all egress. Workflow journeys are a separate operator lane and are never mixed into a hosted scan.
Prisma 7.10.0 has a local Docker adapter. GitHub CLI has a credentialed operator adapter. Neither is a published campaign result. The other nine catalog CLIs are reviewed specifications that still need dedicated evaluation accounts. Local Prisma controls are not agent n/3.
Publication requires 11 complete reports, 99 reference-model journeys inside the $20 ceiling, independently checked state, and a five-reader comprehension test. Missing evidence stays Not measured, never 0/3.
Publication
All 11 reports need complete applicable coverage, verified controls, reviewed task packs and findings, redacted retained logs, and reproducible package and environment identity. The report, JSON, Markdown, and share card use one data projection.
A publication is immutable. A failed refresh keeps the previous valid release and its original date. A release also requires cleanup and monitoring evidence and a five-reader comprehension test.
Historical cold-probe methodology →