<- Research notesAction Theory Lab

RESEARCH / AGENT TRAINING

Building the Experience Loop

How parallel proposal agents, isolated execution, independent verification, and persistent memory can turn authorized defensive tasks into better SFT data, RL rollouts, and evaluations.

Sankalp SinghJuly 21, 202610 min read

A defensive agent should not forget what happened after the verifier returns a score. Every authorized attempt can become evidence, training data, or a new test if the system is designed to retain the right things.

Most agent pipelines throw away their most valuable output.

They keep the final score and discard the path that produced it: the first wrong hypothesis, the tool call that exposed a missing permission, the patch that passed functional tests but left the vulnerability reachable, and the recovery step that finally worked. The next rollout begins with almost none of that experience.

That is acceptable when the objective is a benchmark number. It is wasteful when the objective is to build a dependable system.

Tencent's recently published Hyra scaffold offers a useful pattern. A context agent maintains an Experience Bank, multiple proposal agents explore candidate solutions, fresh sandboxes execute them, and evaluators return scores and artifacts to the bank. The framework is deliberately simple: use search, execution, feedback, and memory rather than encoding every research strategy in the scaffold.

We think this pattern can be adapted to defensive cyber training. The important adaptation is not to place a cyber label on the same loop. It is to make authorization, containment, independent verification, and evaluator integrity part of the loop's definition.

A hand-drawn experience loop in which successful and failed attempts return evidence to a shared research notebook.
Experience becomes useful when verified outcomes return to the next attempt.

The useful part is the loop around the model

Hyra is easy to misunderstand as a model that repeatedly rewrites its own weights. That is not the essential mechanism.

The base model may remain fixed while the surrounding system improves the artifacts it produces. It searches over programs, configurations, training recipes, mathematical constructions, or other candidate solutions. Execution provides evidence. The Experience Bank makes that evidence available to later attempts.

For a history of experiments Ht, a context function C constructs a bounded inspiration packet. Proposal worker j then samples a candidate:

xt+1(j)∼π ⁣(⋅∣C(Ht), g, A)x_{t+1}^{(j)} \sim \pi\!\left(\cdot \mid C(H_t),\,g,\,A\right)

Here g is the task and A is its authorization scope. The candidate is executed and independently evaluated. Its actions, observations, artifacts, score, safety events, and environment version return to Ht+1.

This is agentic search with persistent experimental memory. It can produce material for later model training, but it is not itself PPO, GRPO, or another policy-gradient algorithm.

A defensive loop starts with authorization

An open-ended research agent benefits from a broad action space. A defensive cyber agent needs a broad authorized action space.

Every task entering the loop should therefore carry a signed authorization object containing:

  • target identities and environment IDs;
  • permitted and prohibited actions;
  • network and data-handling boundaries;
  • credential scope and lifetime;
  • tool, token, turn, cost, and wall-clock budgets; and
  • approval requirements for sensitive transitions.

The context agent may decide which prior experiments are relevant. It may not expand the task's authority. Proposal agents inherit the same immutable scope, and the runtime enforces it without relying on the model's interpretation.

Architecture of the proposed cyber experience loop with authorization, parallel proposals, tiered execution, verification, monitoring, and an Experience Bank.
The model proposes. The environment produces evidence. Independent systems decide what becomes experience.

The loop has seven parts:

  1. Authorized task. The objective, scope, environment, and budget are fixed before exploration begins.
  2. Context agent. It selects useful failures, successful techniques, logs, patches, and verifier feedback from the Experience Bank.
  3. Inspiration queue. It holds diverse, bounded packets so proposal workers do not all repeat the same approach.
  4. Proposal agents. Several copies of the current policy produce candidate trajectories and artifacts asynchronously.
  5. Tiered execution. Low-risk repository tasks may use gVisor; stateful services may use Firecracker microVMs; higher-risk or multi-host tasks require isolated VM ranges.
  6. Independent verification and monitoring. One service checks the technical end state. Another checks authorization and the trajectory used to reach it.
  7. Experience Bank. It stores enough immutable evidence to reproduce the attempt and decide how, or whether, it should be reused.

One attempt, three assets

A verified rollout is useful beyond its immediate score.

1. Grounded SFT examples

A successful trajectory can become supervised data after cleaning and review. The retained example should include the task, scope, relevant observations, necessary tool calls, recovery steps, final response, verifier evidence, and provenance.

The best SFT example is not always the shortest successful path. A trajectory that encounters a realistic tool error and recovers correctly may teach more than a perfect first attempt. Conversely, repetitive search, irrelevant logs, accidental secrets, and unsupported reasoning should be removed or masked before training.

Failed attempts are also useful, but usually not as raw imitation targets. They can be transformed into contrastive examples, error diagnoses, corrected continuations, or demonstrations of when the agent should stop and request approval.

2. RL rollouts

The same environment can provide on-policy trajectories after the SFT checkpoint is stable. The reward should separate technical success from authorized execution:

R=IauthorizedIcontained(wsRstate+weRevidence+wrRrecovery−wcCresources)R = I_{\mathrm{authorized}} I_{\mathrm{contained}}\left(w_sR_{\mathrm{state}} + w_eR_{\mathrm{evidence}} + w_rR_{\mathrm{recovery}} - w_cC_{\mathrm{resources}}\right)

The multiplicative gates are deliberate. A technically correct result receives no task reward when it escaped containment or exceeded authorization. This prevents task success from compensating for a system-level safety failure.

3. Regression evaluations

Every unexpected failure should be considered for the private evaluation suite.

If a model reaches the right score by copying a hidden patch, the next environment needs a canary for reference-answer access. If it disables a security control to make tests pass, the verifier needs a state-delta check. If it repeatedly searches for alternative credentials after denial, the trajectory monitor needs an incident-derived evaluation.

The loop therefore connects training and evaluation without merging them. Training may learn from the failure pattern; the frozen test instance and hidden answer must remain excluded.

The verifier is the bottleneck

Search finds whatever the evaluator rewards, including shortcuts its designers did not anticipate.

Tencent reports that Hyra candidates found ways to obtain attractive scores without making genuine progress. One modified a causal language model in a way that leaked future information. Another cached outputs during correctness checks and avoided the intended timed computation. These are useful examples of a general rule: stronger search places more pressure on the evaluator.

For defensive tasks, the same failure could appear as:

  • suppressing a vulnerable test instead of repairing the vulnerability;
  • deleting an alert rather than resolving the incident;
  • broadening an IAM policy to make an application work;
  • hard-coding a known flag or expected artifact;
  • changing telemetry so the verifier can no longer observe the failure; or
  • producing a persuasive report without changing the underlying state.

The verifier should inspect independent state whenever the outcome is mechanically observable. An LLM judge can assess report clarity or the quality of an uncertainty statement. It should not replace a functional test, policy evaluation, packet trace, service-health check, or hidden security regression.

Evaluator evolution needs a protected boundary

Hyra's bilevel loop can improve the solution and then improve the evaluator. That is powerful, but a cyber training system cannot let the same agent silently redefine success.

We separate three objects:

  1. Development evaluator: fast, visible, and editable; used for dense feedback during exploration.
  2. Frozen hidden verifier: immutable within a run; contains hidden tests, canaries, state checks, and authorization assertions.
  3. Promotion process: human-reviewed and versioned; evaluator changes must pass historical incidents and untouched canaries before becoming authoritative.
Evaluator changes crossing hidden canaries and independent review before promotion.
The agent may propose a better measurement. It may not move the release bar by itself.

The Experience Bank is more than an answer cache

Retrieving the nearest successful response would encourage copying and contamination. The bank should preserve experimental structure rather than act as an answer cache.

FieldWhy it matters
task_id, family, difficultyCurriculum and stratified evaluation
authorization_digestProof that the attempt remained within the intended boundary
model_digest, policy_versionExact policy that generated the trajectory
environment_digest, verifier_versionReproduction and invalid-environment analysis
Actions and observationsTool behavior, recovery, and trajectory monitoring
Artifacts and state deltasIndependent evidence of what changed
Task, safety, and cost scoresMulti-objective selection rather than one scalar headline
Failure classificationCurriculum generation and regression routing
Reuse permissionsWhether the record may enter SFT, RL replay, evaluation, or none

The context agent should retrieve abstractions and evidence under contamination controls. Held-out evaluation solutions, private flags, reference patches, and secret verifier behavior must never become inspirations.

Start with secure-code remediation

The first pilot should be deliberately narrow. Secure-code tasks have executable tests, repository state, clear artifacts, and relatively inexpensive resets.

We would begin with:

  • 300-500 owned or permissively licensed repository environments;
  • one vulnerability and one explicit authorization object per task;
  • four to eight proposal rollouts per task;
  • fresh microVM state for every attempt;
  • functional tests, hidden security tests, and patch-minimality checks;
  • one frozen checkpoint with no gradient updates during the initial experiment; and
  • complete trace, cost, environment, and verifier versioning.

The first question is not whether recursive search sounds impressive. It is whether the loop beats a matched single-rollout baseline.

We would measure:

  • pass@1 and success@k;
  • marginal verified gain per additional proposal;
  • cost and wall-clock time per accepted solution;
  • environment-invalid and verifier-error rates;
  • diversity among candidate approaches;
  • reward-hacking discoveries;
  • proportion of trajectories eligible for SFT or RL; and
  • performance on a held-out repository set after any subsequent training.

Only after this loop is stable should it expand to detection engineering, cloud and identity policy, incident response, or multi-host ranges.

What could still go wrong

The architecture does not create trustworthy experience automatically.

Proposal workers share a model and may produce highly correlated mistakes. The Experience Bank can amplify a bad assumption across future attempts. A context agent can leak held-out information. Verifiers can overfit to familiar shortcuts. Sandboxes can fail, leaving invalid trajectories that look like model failures. A large search budget can manufacture one impressive result while hiding poor pass@1 reliability.

These are reasons to version every component, preserve denominators, run multiple seeds, keep private temporal holdouts, and publish failed-environment rates. They are also reasons to begin with a fixed policy and prove the harness before attaching an expensive RL trainer.

The point is accumulated evidence

The durable advantage of a defensive agent will not come from one checkpoint or one benchmark score. It will come from a system that can convert authorized work into reproducible evidence, distinguish genuine progress from shortcuts, and reuse experience without contaminating its own evaluations.

The experience loop gives us a way to connect those pieces:

authorized task -> diverse proposals -> isolated execution -> independent verification -> reusable experience -> better data, training and tests

That is a narrower claim than recursive self-improvement. It is also a more useful engineering objective. Before asking an agent to improve itself, build a system that can tell when an improvement is real.

Further reading