<- Research notesAction Theory Lab

RESEARCH / SFT DESIGN

Measuring What the Model Actually Trains On

Why assistant-loss tokens, curriculum, and held-out capability matter more than raw example counts when building a cybersecurity SFT mixture.

Sankalp SinghJuly 17, 20268 min read

A training row is a storage unit. A loss token is a learning unit. Confusing the two is an easy way to build the wrong model on a perfectly clean dataset.

We audited 142,000 cybersecurity SFT records and found no malformed JSON, empty assistant responses, cross-file exact duplicates, or broken tool-call structures. The corpus was unusually clean for synthetic data.

The row count was still almost useless for deciding how to train on it.

One short reporting answer and one 30-turn tool trajectory both count as one row. They do not contribute the same number of supervised tokens, gradient mass, or behavioral pressure. A row-balanced sampler can quietly turn into a tool-grammar curriculum while the dashboard insists every domain is equal.

This is how we are converting a clean collection of files into a controlled SFT mixture: count the tokens that receive loss, set domain targets in that unit, introduce long trajectories gradually, and keep general capability anchors in every stage.

A technical illustration of supervised fine-tuning data flowing through token accounting, curriculum design, and capability checks.
The training signal is shaped by supervised tokens, not row count alone.

Rows measure files, not training pressure

Suppose two domains each contain 10,000 examples. The first has short classification and reporting targets averaging 300 assistant tokens. The second contains tool calls, observations, retries, and closure reports averaging 1,800 assistant tokens. Sampling the domains equally by row gives the second domain roughly six times as many supervised tokens.

That might be intentional. Usually it is an accident.

The actual SFT objective makes the relevant unit explicit:

LSFT(θ)=−1M∑i,tmi,tlog⁡pθ(yi,t∣xi,yi,<t),M=∑i,tmi,t\mathcal{L}_{\mathrm{SFT}}(\theta) = -\frac{1}{M} \sum_{i,t} m_{i,t} \log p_{\theta}(y_{i,t} \mid x_i, y_{i,<t}), \qquad M = \sum_{i,t} m_{i,t}

The mask mi,t is one only when token t should train the assistant policy. System prompts, user packets, tool observations, padding, and intentionally masked erroneous actions should normally provide context without receiving assistant-policy loss.

That distinction matters. The current TRL SFT trainer supports assistant-only loss, but it depends on a compatible chat template that marks generation spans. It also supports packing several examples into a sequence. Both make file-size estimates a poor proxy for the mixture.

Our audit has about 36.5 million provisional target tokens

Our seven-file allowlist contains 374.40 million characters across system, user, tool, and assistant messages. Dividing that number by four produces a tempting estimate of roughly 94 million tokens. That is not the supervised target.

The assistant responses contain 145.86 million characters. The same rough character proxy gives about 36.5 million assistant-target tokens before accounting for the exact tokenizer, chat template, truncation, special tokens, and action masks.

CorpusRowsAssistant charactersRaw target shareMean per row
Reporting and instruction12,00013.68M9.38%1,140
Secure code25,00026.64M18.27%1,066
SOC and incident response15,00016.00M10.97%1,066
Systems, cloud, and identity25,00026.02M17.84%1,041
Defensive tool use20,00021.14M14.49%1,057
Long-horizon trajectories15,00027.13M18.60%1,809
General anchors30,00015.25M10.45%508

The long-horizon file has half as many rows as the anchor file, but almost 1.8 times as many assistant characters. A row-based chart makes anchors look dominant. A target-token ledger shows the opposite pressure.

These numbers are audit evidence, not final tokenizer counts. Before training, render every record through the exact base-model chat template and count only tokens whose labels are not -100.

A token ledger tracking assistant-loss tokens across SFT domains and comparing requested with realized mixture shares.
A live token ledger makes the intended mixture observable during training.

Solve for token share

Let qd be the probability of drawing a row from domain d, and let mud be the average number of assistant-loss tokens in a sampled row after formatting, masking, and truncation.

τd≈qdμd∑jqjμj\tau_d \approx \frac{q_d \mu_d}{\sum_j q_j \mu_j}

If the desired target share is taud*, a useful initial sampler is qd proportional to taud* / mud. Packing efficiency, sequence length, distributed token averaging, and dropped or truncated samples can move the realized share, so the loop must emit a live ledger of loss tokens by domain.

Our provisional S1 target is:

DomainTarget shareWhy it is present
Reporting and instruction7%Evidence, uncertainty, escalation, and concise closure
Secure code20%Repository reasoning, minimal patches, and regression tests
SOC and incident response13%Telemetry interpretation and containment decisions
Systems, cloud, and identity16%Configuration review, rollback, and ownership boundaries
Defensive tool use14%Schemas, bounded queries, recovery, and valid stopping
Long-horizon trajectories20%State tracking, feedback, retries, and verification
General anchor replay10%General coding, reasoning, instruction following, and safe communication

This mix is not proportional to available rows, and it is not a claim that 20% secure code is optimal. It is a testable configuration with a reason behind every percentage.

Long trajectories belong late

Long-horizon data teaches how actions change state, but it is repetitive. Thousands of trajectories may share the same tool envelope, error format, and closure grammar. Expose that distribution too early and the model may learn scaffolding while getting worse at the underlying engineering work.

Our curriculum changes the sampler over time. General anchors remain active throughout; they are not a warm-up file that disappears once cyber specialization begins. Tool and trajectory shares increase only after held-out checks show valid schema use, grounded closure, safe stopping, and retained general coding behavior.

A staged SFT curriculum that increases tool-use and long-horizon trajectory exposure while retaining general capability anchors.
Long trajectories should enter through a measured curriculum, not a sudden mixture switch.

Quality cannot be repaired with a multiplier

The token ledger controls exposure. It does not make weak data stronger.

  • Reporting repeats a narrow family of privileged-sign-in and approved-change scenarios.
  • SOC data is a workable pilot curriculum, not a substitute for noisy SIEM timelines and tested detections.
  • Secure-code data is the strongest specialist pack, but it still uses small repository slices and repeated weakness families.
  • Tool and long-horizon records contain valid action-observation sequences, but many task graphs remain scripted.
  • General anchors are short and pattern-limited, so 10% replay is minimum protection, not proof against forgetting.

Increasing any file from 25,000 to 250,000 rows with the same templates would make the dataset larger and the model more certain about the same narrow patterns.

The next useful additions are smaller verified packs: repositories with real build output, SOC cases with positive and negative event sets, cloud tasks spanning multiple resources, and trajectories with hidden state verifiers. LIMA shows how carefully selected instruction data can move behavior with surprisingly few examples. Tulu 3 shows why an open post-training recipe needs curation, decontamination, and multi-task evaluation.

Track exposure and capability

We will not approve an S1 checkpoint because it consumed the planned percentages. Exposure is an input. Capability is the result.

  • Requested and realized assistant-loss tokens by domain
  • Truncated, masked, packed, and dropped tokens
  • Sequence-length distribution and packing utilization
  • Held-out domain success and general-capability retention
  • Valid tool-call, safe-stop, and evidence-grounding rates
  • Tokenizer, chat-template, dataset, sampler, and checkpoint digests

mixture_unit: assistant_loss_tokens is the contract the sampler must satisfy and the audit must verify. The exact tokenizer revision and chat-template digest belong beside every checkpoint.

When the pilot disagrees

The first mixture will be wrong in some way. If secure-code performance rises while general coding falls, increase anchor quality or replay, not simply row count. If tool validity improves but task completion does not, inspect whether tool syntax is crowding out planning and evidence interpretation. If long-horizon loss drops while held-out success stays flat, look for memorized trajectory grammar.

Change one pressure at a time and rerun the relevant held-outs. Otherwise a mixture change, curriculum change, and optimizer change collapse into one experiment with no useful conclusion.

Rows remain useful for storage, review queues, and provenance. They are just the wrong denominator for deciding what the model learns.

Further reading