Evaluating Reliability Beyond Pass@k

A successful trajectory proves that the system can solve one sampled task under one harness. Deployment requires evidence that it solves the right tasks repeatedly, stays inside scope, and fails safely when the environment stops cooperating.

An agent patches a vulnerability, runs the tests, and produces a clean report. The verifier returns success. That is useful evidence. It is not a release decision.

The same checkpoint may fail the next seed, call the wrong tool when a schema changes, over-refuse an authorized task, or reach the correct final state through an action the policy gate should have blocked. A single pass hides these failure modes behind one green check.

Our evaluation unit is therefore not the model alone. It is a versioned system: checkpoint, system prompt, tool schemas, harness, memory policy, authorization gate, environment, verifier, retry budget, and inference settings. Change one and you changed the system being evaluated.

Define the claim first

“We scored 47%” is not a claim. It is a number looking for context.

An evaluation should state which question it answers:

1. Capability: Can the system complete a defined defensive task under a credible tool and budget configuration?
2. Comparison: Does one checkpoint or harness outperform another under matched conditions?
3. Safeguard performance: Do the surrounding policy and runtime stop disallowed behavior while permitting authorized work?
4. Release readiness: Does the candidate meet capability, safety, reliability, and operational thresholds for a deployment boundary?

The harness should match the claim. A shell loop may be appropriate for controlled model comparison, but it is not evidence about a deployment that uses memory, retries, policy gates, and a different tool interface. OpenAI’s evaluation playbook makes the same point: reports should identify the claim, system, tools, budget, elicitation method, and checks for reward hacking, contamination, refusals, broken tasks, and evaluation awareness.

Pass@k measures possibility

For task i, let si,j indicate whether attempt j reached the verified objective. A common success-at-k statistic is:

This asks whether the system succeeded at least once. It is useful for estimating a capability ceiling under repeated elicitation. It can also flatter an unreliable agent.

For deployment, we additionally care whether a task succeeds consistently. For required per-task reliability rho:

If k = 5 and rho = 0.8, a task counts as reliable only when at least four attempts succeed. Publish both metrics with confidence intervals. Also report pass@1: a system that needs eight retries to look competent may be useful for research, but it is expensive and unpredictable in production.

Separate model and infrastructure failure

Agent evaluations fail in at least three places:

• Policy failure: the model chooses the wrong action, misreads evidence, or stops early.
• Harness failure: state is dropped, a tool result is malformed, or context management removes required evidence.
• Environment failure: reset breaks, a service never starts, storage is exhausted, or the verifier reads stale state.

Score capability only on tasks that pass environment-validity checks, and publish invalid-environment rate separately:

The denominator must remain visible. Quietly dropping 12% of tasks because difficult environments were flaky is another way to manufacture a nice score.

Every task needs preflight and post-run checks: services healthy, hidden state present, objective reachable, clocks and credentials scoped, verifier version matched to the image, and teardown complete.

Use five release gates

Our release decision is a conjunction:

If any term is zero, the candidate does not cross the proposed deployment boundary. Strong patching performance cannot compensate for an authorization bypass; perfect containment cannot compensate for a model that never completes the task.

Thresholds depend on deployment scope. A read-only triage assistant and an agent allowed to change production identity policy should not share a release bar.

Measure attempts and executions

The harness should block unsafe actions independently of the model. That creates two signals:

1. Unauthorized action attempt rate: how often the model asks to do something outside the signed scope.
2. Unauthorized action execution rate: how often controls allow such an action to reach the runtime.

The first measures model behavior. The second measures a system safety failure and should be zero for release. Also track unnecessary refusal, escalation quality, and approval calibration. Refusing every cloud change produces a low violation rate and no operational value.

The gate is part of the system under test, but it must not depend solely on the model’s explanation of what it intends to do.

Benchmarks are comparison points

Public benchmarks give shared language and external comparability. They cannot cover internal tool schemas, authorization models, enterprise state, or failure modes.

• Cybench provides 40 professional-level CTF tasks with an agent harness and subtasks.
• CyberGym contains 1,507 historical vulnerabilities across 188 software projects.
• CyberGym-E2E extends discovery, proof-of-concept generation, and patching.
• SecRepoBench tests secure generation across real C and C++ repositories.
• CyberSOCEval covers malware analysis and threat-intelligence reasoning.
• SIR-Bench evaluates incident-investigation depth and tool appropriateness.

Primary release signals should come from private, temporally held-out, executable tasks built after the training cutoff. Public instances, reference patches, flags, and near-duplicates belong in an exclusion registry, not in SFT or RL replay.

Verify the end state

An agent can produce a persuasive report while failing the task. It can also solve the task while using an unintended shortcut. The verifier should inspect independent state:

LLM judges can score report clarity, but they should not replace a deterministic check when the final state is observable. Hidden tests must remain hidden from both model and training pipeline. Include negative tests and canaries for hard-coded flags, reference-patch copying, grader manipulation, and destructive shortcuts.

Version the harness

A checkpoint name is not a reproducible evaluation configuration. Every result needs model and tokenizer digests; system prompt and reasoning configuration; tool schemas, permissions, and policy rules; memory, compaction, retry, and stopping behavior; environment and verifier versions; token, turn, wall-clock, and cost budgets; sampling parameters and seeds; and task inclusion, invalidation, and denominator rules.

If any component changes, create a new evaluation configuration instead of silently appending results to the old one.

Publish the whole result

Every serious candidate report should contain:

1. The exact claim and deployment boundary.
2. Task counts and distributions before and after invalidation.
3. Pass@1, Success@k, Reliable@(rho,k), confidence intervals, and per-domain results.
4. Environment failure, timeout, malformed-action, and recovery rates.
5. Unauthorized attempts, blocked actions, executed violations, and over-refusals.
6. Token, turn, wall-clock, tool, and inference cost per attempt and successful task.
7. Contamination, reward-hacking, broken-task, and verifier-integrity audits.
8. Failure clusters with representative traces and reproducible versions.

We will not hide failed environments, choose the best seed after seeing answers, or report pass@k without pass@1. The useful question is whether a named version of the whole system completes a defined class of authorized tasks repeatedly, under a budget, while independent controls contain its mistakes.

Further reading

• A shared playbook for trustworthy third-party evaluations
• Cybench
• CyberGym
• CyberGym-E2E
• SecRepoBench
• CyberSOCEval
• SIR-Bench