Your Cyber Agent Does Not Learn Security From a Dataset

You can fine-tune a model on 100 million tokens of security material and still get an agent that confidently writes a broken patch, misreads an alert, or changes the wrong cloud policy.

The problem is not that the model has never seen a good answer. Static examples give it the answer before it has to deal with the consequences of an action.

That matters for agents. An agent has to inspect a state, pick a tool, interpret an observation, recover when the tool call fails, validate the result, and stop when its authorization ends. A JSONL record can demonstrate that sequence. It cannot prove the model can execute it.

This is the training architecture we are building toward: SFT for foundations, resettable environments for consequences, reinforcement learning for behavior, and a deployment harness that can say no even when the model says go.

SFT teaches the agent the grammar of work, not whether the work succeeded

SFT is still the right place to teach the model how a defensive agent should behave before it gets near an environment.

It needs to understand code and systems. It needs to know how to write an incident report that separates evidence from speculation. It needs to emit valid tool calls, interpret a configuration file, and recognize that rotating every credential is not a sensible default response to an alert.

That is a large job. It is also not the same job as agent training.

Consider a secure-code example. A static record can show a repository, a vulnerability description, a patch, and a passing test. The model can learn the shape of a good answer.

But it has not had to locate the relevant file, choose between two plausible patches, run the tests, see a failure, revise its approach, and prove that the patch fixed the problem without breaking the service. That is the part we care about when we hand it tools.

The model should see both. Treating one as a substitute for the other is how you end up measuring elegant prose instead of useful work.

The loop we want is task, action, observation, verification

The basic loop is deliberately boring. Boring is good here.

The policy sees the task and proposes what to do. An authorization and policy gate decides whether the action may reach a tool runtime. The environment executes only approved actions and returns observations. The verifier checks the final state independently of the model's explanation.

That last part is non-negotiable. A model should not receive credit because it wrote a convincing paragraph saying the patch worked.

For a defensive configuration task, the verifier might check that a policy changed, that the service remains healthy, and that the action stayed inside its signed scope. For repository work, it might run tests and inspect the final diff. For an investigation task, it might validate the evidence chain and the required case state.

The model is allowed to be wrong. The environment is not allowed to be vague about whether it was right.

The verifier, not the model, decides whether a trajectory earned reward

In a good agent environment, the reward hierarchy is simple:

Here, Iauthorized and Icontained are binary gates. Rtask and Revidence are normalized scores in [0, 1]. If an action was unauthorized or escaped the environment boundary, the reward is zero. An escape is not merely a zero-reward sample: its trace must be quarantined, the environment must be torn down, and the incident must be investigated.

The rest rewards the task outcome first, then the quality of the evidence. Efficiency can matter later, once correctness is established. Rewarding speed before correctness is how an agent learns to end a task quickly instead of finishing it well.

This is why we prefer executable tests, state-delta checks, and structured validators over LLM judges whenever possible. A judge can help assess report quality or explain a borderline decision. It should not be the sole authority for an outcome that a program can check.

Why we are not starting with one enormous cyber RL run

Combining every domain, reward, and tool in one run makes regressions hard to explain. If repository patching improves while identity decisions worsen, the cause could be the data mix, reward, environment, optimizer, or harness.

Start with environments where success is executable and obvious:

1. Secure repository remediation. Find, patch, test, and verify a controlled issue.
2. Authorized configuration validation. Inspect a controlled system or cloud-like policy, make a minimal approved change, and confirm policy state and service health.

These give useful failure traces. Add SOC workflows, network telemetry, forensics, and cross-resource identity only after the environment/verifier loop is trustworthy; scaling a weak fleet just multiplies ambiguity.

Long-horizon RL needs a different kind of plumbing

SFT examples are files; agent RL produces living traces:

task
  -> inspect
  -> tool call
  -> observation
  -> diagnosis
  -> corrective action
  -> validation
  -> evidence-backed report

Traces may take four turns or 200; resets take seconds, while tests and sandbox initialization take minutes. Asynchronous rollouts, policy lag, and value estimation matter because waiting for the slowest task leaves the fleet idle.

GRPO compares several rollouts for one prompt without a learned critic, which suits short, cheap tasks. Actor-critic methods use a value estimate to judge partial progress, trading extra model cost for fewer rollouts. SAO is one recent proposal that uses rollout log-probabilities and rejects updates that drift too far from the producing policy. DeepSeekMath SAO

On-policy training uses current-policy trajectories; PPO controls the probability ratio, while GRPO uses relative rollout comparisons. Off-policy training reuses replay from an older behavior policy, improving sample efficiency but requiring distribution-shift controls such as importance sampling, conservative objectives, and policy-version metadata.

A hybrid is plausible here: fresh on-policy rollouts provide trustworthy task and safety rewards, replay teaches recovery and rare failures, and actor-critic estimates with Generalized Advantage Estimation (GAE) provide denser credit. Every trace still needs behavior-policy, environment, tool-scope, and verifier versions.

The training system has more moving parts than the model

The model is the visible part. The system that makes its behavior measurable is the important part.

The task router assigns work. Rollout workers run the model and tool loop. The environment fleet returns controlled observations. The verifier records whether the task actually completed. The trainer updates the policy from trajectories that are still valid for the current training step.

A critic and a routed teacher solve different problems. In actor-critic RL, the critic estimates expected future return from a partial trajectory. In on-policy distillation, a selected teacher scores the student's sampled tokens. A run may use one or both, but they should not be treated as interchangeable signals.

The trace store is not an afterthought. Each record needs task, environment, policy, reward, verifier, tool, and artifact versions. If we cannot reproduce a surprising result, it is not a research result yet.

Safety cannot be another item in the prompt

The model can recommend an action. It cannot be the final authority on whether that action is allowed.

Our defensive agent design has independent controls around the policy:

This is not a substitute for model training. It is how we make model mistakes survivable.

We will also keep internal red-team capability separate from the deployed defensive agent. A red-team policy can create adversarial test cases in controlled simulations. It should not be shipped as a product feature because someone put "for authorized use only" in the README. That is not a control.

A benchmark score is not a release decision

We will report benchmark results when they are useful. We will not treat them as proof that the system is ready for real work.

For every evaluation, we need to disclose the full setup:

• the claim being tested;
• checkpoint, inference settings, and tool permissions;
• harness and environment version;
• task distribution and verification method;
• token, turn, retry, and wall-clock budget;
• failure rate of the environments themselves;
• contamination, reward-hacking, and broken-task checks; and
• safety violations and over-refusals.

This is less glamorous than publishing one large number. It is also the only way another engineer can decide whether the number applies to their system. Evaluation reports for agentic systems need to describe the harness as well as the model because the harness changes what the model can do. A shared playbook for trustworthy third-party evaluations

What we are building first

We are starting with a smaller claim and a higher bar for evidence.

1. Freeze one cyber SFT checkpoint and its general-capability baselines.
2. Build one environment and verifier SDK with deterministic reset, teardown, and trace identity.
3. Run secure-repository remediation and authorized-configuration pilots.
4. Compare SFT-only behavior with a small, controlled RL experiment.
5. Expand only when task success, safety retention, and environment reliability are measured together.

The first useful result may be boring: a small model improvement, a reward that was easy to game, or an environment reset bug that invalidated a week of rollouts. We want to find those problems while the system is small enough to understand.

The goal is not to make a model sound like a security expert. The goal is to build a system where correct, authorized defensive work is observable and repeatable.

Further reading

• Proximal Policy Optimization Algorithms
• Generalized Advantage Estimation
• DeepSeekMath and Group Relative Policy Optimization
• Single-Rollout Asynchronous Optimization for Agentic RL
• A shared playbook for trustworthy third-party evaluations