Steered paraphrase pretraining: what changes, and where the gains come from
Small LMs pretrained on FAQ paraphrases generated with activation steering (register, wiki and new contrast directions). Results summary, updated 2026-11-20 with the new arms and both combined vectors at both model sizes (§8) and a pick of the best vector so far (§11); every number below is computed from the CSVs in data/.
Summary
- New (§8): every steered corpus beats base at both model sizes, and the faithfulness-corrected vector is the best arm so far.
reg_plus_faithα39 gains +1.42 (Qwen2 1.7B) and +0.98 (2.9B) macro points, ahead of register α35 at both sizes, and lowers held-out loss in all 7 domains (+2.34 / +2.72 centi-nats). Addingedu_qualityto it gives the largest PMI-accuracy gains (+2.44 / +3.11 points) and science-knowledge gains (ARC) at a cost on tables; halving the edu weight (rpf_plus_edu50, +1.33 / +0.89) keeps half of the knowledge gain and removes most of that cost. Best so far: reg_plus_faith, with rpf_plus_edu50 a close second (§11). With scale, cloze gains shrink by a third to a half while accuracy-type and loss gains hold. - Benchmarks (register and wiki arms): macro changes from +0.01 to +1.03 points (Qwen2 1.7B register α35; Qwen2 2.9B register α35 +0.71). Register leads wiki in all five comparisons. Gains on DROP, tables, XCSQA, GSM8K (Qwen); costs on LAMBADA and SQuADv2.
- Probability vs accuracy: at 1–1.7B the gain is mostly higher probability on answer-like text; at 2.9B it is more correct choices. Knowledge is unchanged by register and wiki (edu_quality changes that, §8).
- Language modelling: every steered model lowers held-out loss by ~1–2% perplexity, wiki most and α35 least: the reverse of the benchmark ranking.
- Mechanism (loss): gains on hard/rare tokens and layout; α35 gives up gains on function words and prose.
- Mechanism (benchmarks): gains are on the gold answer, not on reading the context. Steered models prefer short span answers, most at α35, and this roughly tracks the benchmark gains.
- Data: steering makes FAQ answers open more directly and contain more numbers that are not in the source document; register and wiki data are nearly identical.
- Vectors: more human sources give the same direction (cos 0.91–0.92 with register) with more cost. New contrasts are new directions; random directions do nothing to the text; register + a faithfulness correction removes the unsupported numbers while keeping the NLL shift, and all three new directions improve benchmarks when trained at full scale (§8).
- Generation: 3 vLLM engines per GPU, ~1.2× faster with the same output distribution.
Setup
- Models: Llama 1B and Qwen2 1.7B (base + four steered models each) and Qwen2 2.9B (base, register α35, wiki α35), pretrained from scratch, one seed each. New in §8: reg_plus_faith α39, edu_quality α35, formulaicity α9 and the combined rpf_plus_edu100 α39 and rpf_plus_edu50 α39 at Qwen2 1.7B and 2.9B.
- Data: one epoch over a 20B-token mix, 50:50 synthetic and human (FineWeb). The synthetic half is FineWeb rewritten as FAQs by SmolLM2-1.7B-Instruct. Base: no steering. Register and wiki: a steering vector added at layer 14 with strength α = 15 or 35.
- Units: benchmark Δ in points (cloze score, prob_norm_token × 100) or accuracy percentage points (§8); loss Δ in centi-nats (nats × 100), positive = better than base. CIs are 95% paired bootstraps.
1 · Benchmarks (register and wiki): small gains; one clearly above seed noise
Paper-style evaluation (lighteval, 3-shot, cloze, prob_norm_token), every question of every task, paired bootstrap against each architecture's base. Shaded band: the finephrase paper's seed-to-seed SD of this score (0.16–0.49 points). One seed per model.
- Largest gain: Qwen2 1.7B register α35 +1.03 [+0.95, +1.11]. Qwen2 2.9B register α35 +0.71, wiki α35 +0.26. Base macro: Llama 16.44, Qwen 1.7B 17.53, Qwen 2.9B 18.66.
- The 2.9B models saw the same 20B tokens (≈7.5 tokens/parameter vs ≈12 at 1.7B), so they are the least trained.
Direct comparisons (macro Δ, points, 95% CI)
| contrast | Llama 1B | Qwen2 1.7B | Qwen2 2.9B |
|---|---|---|---|
| register − wiki, α15 | +0.12 [+0.06, +0.19] | +0.29 [+0.22, +0.36] | – |
| register − wiki, α35 | +0.38 [+0.31, +0.44] | +0.38 [+0.31, +0.46] | +0.45 [+0.38, +0.52] |
| α35 − α15, register | -0.06 [-0.13, +0.01] | +0.37 [+0.30, +0.45] | – |
| α35 − α15, wiki | -0.32 [-0.38, -0.26] | +0.28 [+0.21, +0.35] | – |
Register beats wiki in all five comparisons (widest at 2.9B: +0.45); single-seed margins below ~0.23–0.69 are within the noise expected between two models.
Per task
| WikiTableQ | TREB-QA | DROP | SQuADv2 | TriviaQA | XCSQA | ARC-Easy | ARC-Challenge | OpenBookQA | HellaSwag | PIQA | WinoGrande | MMLU-Redux | GSM8K | LAMBADA | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama 1B · register α15 | +0.50† | +0.09 | +2.13† | +1.50† | +0.17† | +0.88† | +0.56† | +0.40† | +0.18† | +0.15† | +0.23† | +0.39† | +0.28† | -0.55† | +0.24 |
| Llama 1B · register α35 | +2.52† | +0.88† | +2.81† | +0.54† | +0.69† | +1.21† | +0.84† | +0.33† | +0.28† | +0.07† | +0.08† | +0.28† | +0.33† | -2.35† | -0.86† |
| Llama 1B · wiki α15 | +0.39† | +0.94† | +1.51† | +1.36† | +0.53† | +0.70† | +0.54† | +0.31† | +0.12† | +0.04† | +0.25† | +0.57† | +0.06 | -1.11† | -0.26 |
| Llama 1B · wiki α35 | +0.84† | -0.16 | +2.26† | -0.94† | +0.79† | +1.28† | +1.04† | +0.35† | +0.20† | +0.09† | +0.34† | +0.42† | +0.35† | -2.51† | +0.25 |
| Qwen2 1.7B · register α15 | +2.35† | +0.96† | +2.18† | +0.51† | +0.27† | +0.10 | +0.33† | +0.23† | +0.20† | +0.18† | +0.09† | -0.04 | +0.14† | +0.47† | +0.05 |
| Qwen2 1.7B · register α35 | +4.32† | +1.65† | +2.75† | -0.10 | +0.09 | +0.60† | +0.42† | +0.37† | +0.53† | +0.15† | +0.00 | +0.34† | +0.28† | +0.87† | -0.86† |
| Qwen2 1.7B · wiki α15 | +1.90† | +0.91† | -0.71† | -0.96† | +0.15† | +0.56† | +0.48† | +0.30† | +0.38† | +0.23† | +0.12† | +0.04 | +0.13 | +0.80† | +0.53† |
| Qwen2 1.7B · wiki α35 | +3.43† | +1.60† | +2.77† | -1.57† | +0.22† | +0.34† | +0.41† | +0.38† | +0.42† | +0.21† | +0.19† | +0.28† | -0.00 | -0.03 | -0.08 |
| Qwen2 2.9B · register α35 | +1.70† | +0.13 | +4.99† | -1.31† | +0.43† | +1.41† | -0.07 | +0.11 | +0.06 | -0.05† | -0.00 | -0.43† | +0.10 | +1.25† | -1.06† |
| Qwen2 2.9B · wiki α35 | -0.17† | -0.41 | +2.23† | -1.55† | +0.52† | +1.06† | +0.45† | +0.06 | +0.14 | +0.06† | +0.14† | -0.40† | +0.08 | +0.96† | -0.01 |
Δ prob_norm_token × 100 vs base, full evaluation sets. † = significant after BH correction (q < 0.05). Blue = better than base, red = worse. Hover a cell for its value.
- 105 of 150 model × task changes are significantly positive, 17 significantly negative.
- Gains: DROP (significant in 9 of 10 models, up to +4.99), WikiTableQ (9 of 10; -0.17 to +4.32), XCSQA (positive in all 10, up to +1.41), GSM8K at both Qwen sizes, TriviaQA at 2.9B.
- Costs: LAMBADA for register α35 in all three families (-0.86, -0.86, -1.06), SQuADv2 at 2.9B, GSM8K on Llama (down to -2.51). Knowledge tasks (MMLU-Redux, ARC-Challenge, OpenBookQA) barely move.
Probability vs accuracy: what the models actually learn
Only the 8 multiple-choice tasks have an accuracy (acc_norm_token). The two metrics answer different questions: probability = how much mass the correct answer gets; accuracy = whether it beats the wrong options. Probability up with accuracy flat means the wrong options rose too (a general boost to answer-like text). Accuracy up with probability flat means better choices without more confidence.
| task (base acc %: L / Q1.7 / Q2.9) | L reg α15 | L reg α35 | L wiki α15 | L wiki α35 | Q1.7 reg α15 | Q1.7 reg α35 | Q1.7 wiki α15 | Q1.7 wiki α35 | Q2.9 reg α35 | Q2.9 wiki α35 |
|---|---|---|---|---|---|---|---|---|---|---|
| XCSQA (39 / 45 / 47) | +2.9† / +0.88† | +5.9† / +1.21† | +4.0† / +0.70† | +4.5† / +1.28† | +0.1 / +0.10 | +2.8 / +0.60† | +2.7 / +0.56† | +1.4 / +0.34† | +4.4† / +1.41† | +0.8 / +1.06† |
| HellaSwag (43 / 46 / 50) | +1.0† / +0.15† | +0.9† / +0.07† | +0.6 / +0.04† | +0.7 / +0.09† | +0.9† / +0.18† | +1.5† / +0.15† | +0.8 / +0.23† | +1.5† / +0.21† | +1.0† / -0.05† | +1.5† / +0.06† |
| ARC-Easy (52 / 57 / 58) | +1.0 / +0.56† | +1.6 / +0.84† | +2.4† / +0.54† | +2.5† / +1.04† | +1.3 / +0.33† | +0.8 / +0.42† | +1.1 / +0.48† | +1.3 / +0.41† | +3.1† / -0.07 | +3.1† / +0.45† |
| WinoGrande (53 / 52 / 54) | -0.6 / +0.39† | +0.5 / +0.28† | -0.6 / +0.57† | -0.9 / +0.42† | +1.0 / -0.04 | +0.9 / +0.34† | -0.2 / +0.04 | +1.3 / +0.28† | -1.0 / -0.43† | -1.0 / -0.40† |
| PIQA (71 / 72 / 73) | +0.8 / +0.23† | +0.9 / +0.08† | +1.9† / +0.25† | +1.0 / +0.34† | +1.1 / +0.09† | +0.3 / +0.00 | +0.4 / +0.12† | +0.7 / +0.19† | +0.8 / -0.00 | +0.7 / +0.14† |
| ARC-Challenge (26 / 27 / 30) | +0.2 / +0.40† | +1.1 / +0.33† | -1.0 / +0.31† | +0.3 / +0.35† | -0.6 / +0.23† | +1.7 / +0.37† | +0.5 / +0.30† | +1.4 / +0.38† | -0.4 / +0.11 | +0.7 / +0.06 |
| OpenBookQA (26 / 28 / 28) | +0.2 / +0.18† | +0.8 / +0.28† | +0.2 / +0.12† | -0.2 / +0.20† | -1.0 / +0.20† | +2.6 / +0.53† | -1.0 / +0.38† | +1.0 / +0.42† | +0.0 / +0.06 | -0.4 / +0.14 |
| MMLU-Redux (30 / 31 / 32) | +0.3 / +0.28† | +0.4 / +0.33† | +0.2 / +0.06 | +0.8 / +0.35† | +0.1 / +0.14† | -0.1 / +0.28† | -0.3 / +0.13 | +0.3 / -0.00 | -1.0 / +0.10 | +0.0 / +0.08 |
| significant gains / losses, probability | 8 / 0 | 8 / 0 | 7 / 0 | 8 / 0 | 6 / 0 | 7 / 0 | 6 / 0 | 7 / 0 | 1 / 2 | 4 / 1 |
| significant gains / losses, accuracy | 2 / 0 | 2 / 0 | 3 / 0 | 2 / 0 | 1 / 0 | 1 / 0 | 0 / 0 | 1 / 0 | 3 / 0 | 2 / 0 |
Accuracy Δ / probability Δ, points; † = BH q < 0.05. Accuracy needs ~2–3 points to be significant on these set sizes (500–2,400 questions; HellaSwag 10,042).
- Llama 1B and Qwen 1.7B: mostly a format effect. Probability rises on 6–8 of 8 tasks, accuracy on only 0–3 (XCSQA, HellaSwag).
- Qwen 2.9B: better choices. Probability is flat, but register α35 accuracy rises on ARC-Easy (+3.1), XCSQA (+4.4) and HellaSwag (+1.0, while its probability falls).
- The prose cost is lost confidence, not worse judgement (HellaSwag at 2.9B); WinoGrande at 2.9B is the only task where both point down.
- In short: better at turning a question into a short, specific answer; worse at plain-prose likelihood and exact span copying; knowledge unchanged. The size pattern rests on one seed per size.
Sources: aggregates, task deltas, contrasts (and the qwen, qwen29 files); write-up LIGHTEVAL_NALL.md (findings 6–7).
2 · Held-out loss: every model improves, wiki most, α35 least
Per-domain held-out loss, recomputed on the full evaluation splits (the final_eval.json written at the end of training held the FineWeb loss under every domain name). Paired bootstrap over sequences.
| FineWeb | Wikipedia | arXiv | HackerNews | StackExchange | FreeLaw | Gutenberg | Mean | |
|---|---|---|---|---|---|---|---|---|
| Llama 1B · register α15 | +0.62† | +1.38† | +3.70† | +1.73† | +2.90† | +1.08† | +1.63† | +1.86† |
| Llama 1B · register α35 | -0.29† | +1.27† | +2.75† | +2.52† | +2.54† | +0.20† | +0.34† | +1.33† |
| Llama 1B · wiki α15 | +0.32† | +1.78† | +4.80† | +1.74† | +2.99† | +1.13† | +1.49† | +2.04† |
| Llama 1B · wiki α35 | -0.36† | +2.90† | +5.60† | +1.84† | +2.95† | +0.35† | +0.48† | +1.96† |
| Qwen2 1.7B · register α15 | +0.51† | +1.26† | +3.22† | +1.62† | +2.66† | +1.10† | +1.22† | +1.66† |
| Qwen2 1.7B · register α35 | -0.44† | +1.03† | +2.17† | +2.02† | +2.18† | +0.18† | +0.33† | +1.07† |
| Qwen2 1.7B · wiki α15 | +0.40† | +1.69† | +4.20† | +1.11† | +3.10† | +1.33† | +1.54† | +1.91† |
| Qwen2 1.7B · wiki α35 | -0.40† | +2.50† | +4.80† | +1.11† | +2.83† | +0.52† | +0.25† | +1.66† |
Δ loss vs base in centi-nats (nats × 100) on the full held-out splits; positive = lower loss than base. † = BH q < 0.05.
- Mean gain: wiki α15 +2.04 / +1.91 (Llama / Qwen) down to register α35 +1.33 / +1.07 centi-nats, i.e. about 1–2% perplexity.
- The ranking is the reverse of the benchmarks: wiki beats register on loss (register − wiki at α35: -0.63 / -0.59), mainly on arXiv and Wikipedia.
- α35 models are worse than base on FineWeb (-0.44 to -0.29).
- Qwen2 2.9B (from its end-of-training
final_eval.json, not in the table): mean gain register α35 +0.96, wiki α35 +1.43, FineWeb −0.77 for both. The benefit shrinks gradually with size (Llama → Qwen 1.7B → 2.9B).
Sources: ppl_deltas.csv, ppl_contrasts.csv; write-up PPL_FINDINGS.md.
3 · Where the loss gains come from
All 10 models scored on identical tokens (7 domains × 1,024 sequences × 2,048 tokens). Median over the 7 domains of the mean Δ per token group, centi-nats (L = Llama, Q = Qwen):
| token group | L register α15 | L register α35 | L wiki α15 | L wiki α35 | Q register α15 | Q register α35 | Q wiki α15 | Q wiki α35 |
|---|---|---|---|---|---|---|---|---|
| easiest 20% of tokens | -0.02 | +0.06 | -0.03 | -0.04 | +0.04 | +0.03 | -0.10 | -0.12 |
| hardest 20% of tokens | +2.98 | +4.02 | +3.78 | +4.95 | +3.05 | +4.38 | +5.67 | +3.81 |
| tokens unseen in base training corpus | +9.44 | +8.96 | +10.45 | +11.55 | +5.48 | +2.15 | +7.88 | +7.87 |
| 100 most frequent tokens | +0.84 | +0.84 | +0.81 | +0.26 | +0.38 | +0.37 | +0.17 | +0.07 |
| function words | +0.97 | +0.04 | +0.91 | +0.17 | +0.96 | -0.11 | +0.74 | -0.21 |
| whitespace / layout | +7.30 | +10.81 | +5.19 | +12.28 | +5.48 | +9.45 | +6.40 | +10.37 |
- Gains sit on hard and rare tokens and on whitespace; the easiest tokens are unchanged.
- α35 loses the gain on function words (and on words in FineWeb and FreeLaw, below), while its whitespace gains grow.
- On the hardest 20% of tokens, loss falls by 2.0–5.0 centi-nats while predictive entropy rises by at most 1.2 and in 6 of 8 models falls (by up to 2.5; medians over domains). The models rank the right token higher; they don't hedge more.
α35 predicts words worse than base on FineWeb and FreeLaw
| domain | words, α15 models | words, α35 models | whitespace, α35 models |
|---|---|---|---|
| FineWeb | +0.35 to +0.58 | -0.39 to -0.21 | -1.9 to -0.3 |
| FreeLaw | +0.21 to +0.84 | -1.10 to -0.50 | +12.3 to +16.6 |
| Gutenberg | +0.80 to +1.98 | -0.82 to +0.23 | +2.6 to +12.4 |
| arXiv | +3.25 to +4.42 | +1.82 to +5.28 | +9.4 to +17.2 |
Δ on non-whitespace tokens, derived from the total Δ and the whitespace share/Δ ((total − s·Δws)/(1 − s)); ranges over register and wiki in both architectures, centi-nats.
The steered training text is less predictable
NLL change of the synthetic training text vs the base corpus on common tokens, under two outside scorers (negative = less predictable): Qwen2.5-1.5B register α15 -0.11, register α35 -0.28, wiki α15 -0.11, wiki α35 -0.25; Pythia-1.4B register α15 -0.07, register α35 -0.21, wiki α15 -0.07, wiki α35 -0.19 nats. α35 corpora are 2–3× further from base than α15; register and wiki corpora are nearly identical. The less predictable corpora give the smaller loss gains (α35 < α15).
Sources: attribution (llama/qwen), token_overall.csv, token_special.csv, token_calib_quintiles.csv, corpus predictability (qwen/pythia); write-up FINDINGS_all_arms.md.
4 · Benchmark gains are on the answer, not on reading the input
All 10 models were re-scored on the exact lighteval prompts (3-shot) plus the gold answer, for 1,500 items each of WikiTableQ, SQuADv2 and DROP, all 490 TREB-QA, and 2,000 each of TriviaQA and LAMBADA. Our answer log-probabilities reproduce lighteval's per item (correlation ≥ 0.9999).
- The gold answer gains 3.8–11.2 centi-nats on average; the question and passage/table text of the same items gains only 0.6–2.4.
- Across items, how much better a model reads the context does not predict how much better it predicts the answer (|Spearman ρ| ≤ 0.10 in every model × task).
Register vs wiki at α35: better answers, worse reading
| task | Llama 1B: answer / context | Qwen2 1.7B: answer / context |
|---|---|---|
| WikiTableQ | +8.3† / -1.4† | +3.0† / -0.6† |
| DROP | +4.4† / -2.3† | +5.4† / -0.9† |
| SQuADv2 | +6.1† / -2.3† | +1.7 / -1.9† |
| TriviaQA | +4.1† / -2.2† | +3.9† / -2.8† |
| TREB-QA | +7.7† / -0.6† | -1.5 / -0.1 |
| LAMBADA | -8.6† / +0.1 | -4.7† / -0.3 |
Loss difference register α35 − wiki α35 (positive = register lower loss), centi-nats; † = BH q < 0.05.
Sources: bench_answer.csv, bench_segments.csv, bench_contrasts.csv, bench_link.csv, bench_sanity.csv; write-up FINDINGS_benchmark_and_format.md.
5 · What steering changes in the training answers
All synthetic corpora rewrite the same FineWeb documents as FAQs (SmolLM2-1.7B-Instruct, FAQ prompt). 15,000 source documents rewritten by every corpus, compared paired; document-format rows from 20,000 documents per corpus.
| base | register α15 | register α35 | wiki α15 | wiki α35 | |
|---|---|---|---|---|---|
| words per document | 320 | 337 | 416 | 314 | 390 |
| Q/A pairs per document | 5.8 | 6.1 | 6.9 | 6.0 | 6.7 |
| words per answer | 43.6 | 44.7 | 54.0 | 42.5 | 50.7 |
| answer opens with a short sentence (≤ 8 words) | 13.7% | 18.3% | 24.8% | 17.9% | 23.4% |
| answer first restates the question | 44.8% | 44.5% | 39.0% | 43.3% | 38.7% |
compact Q↵A: layout (benchmark style) | 43.6% | 40.4% | 33.1% | 40.5% | 38.4% |
| answer numbers not supported by the source | 19.6% | 27.1% | 41.9% | 27.7% | 43.7% |
| answer numbers that are list numbering | 10.9% | 11.5% | 10.4% | 10.1% | 7.5% |
- Answers get longer, but more of them open with a short direct reply (13.7% → 23.4–24.8% at α35) and fewer restate the question.
- Fewer documents use the compact benchmark layout after steering: steering does not make the data look more like the benchmarks.
- More answer numbers are added by the rewrite (not in the source document, after excluding list numbering, number words, reformatting and numbers from the question): 20% in base → 42–44% at α35. From reading examples: mostly invented details, a few true background facts.
- Register and wiki corpora are almost identical; their differences are 1–3 points:
| paired difference (points) | register − wiki, α15 | register − wiki, α35 |
|---|---|---|
| answer starts with yes/no | -1.4† | -3.1† |
| answer starts with a content word | +0.9† | +2.2† |
| answer opens with a short sentence | +0.5 | +1.8† |
| answer starts with words/number from the source | +1.2† | +1.7† |
† = 95% bootstrap CI over documents excludes 0. Benchmark gold answers are 1–2 words and almost never start with yes/no, so register's direction is closer to them, but the gap is small.
Sources: answers_content.csv, answers_contrasts.csv, answers_numbers.csv, corpus_format.csv; write-up FINDINGS_answer_content.md.
6 · Steered models prefer to answer with a short span
On 2,500 benchmark items (500 per QA task), the probability that the first token after Answer: starts a span (a number, name or word) rather than a sentence ("The", "It", "Yes", …), and the models' own greedy answers. 0-shot = question with its passage/table only; 3-shot = the benchmark prompt.
- Every steered model prefers spans more than its base, in both settings (31 of 32 differences significant, counting span probability and greedy answers). Without demos, base models answer with a sentence 89–91% of the time; α35 models 74–79%.
- Register > wiki in 4 of 4 comparisons at 0-shot, but only 1 of 4 at 3-shot.
- The preference roughly tracks the benchmark gains across the 8 models (Spearman ρ: P(span) 0.81 at 0-shot, 0.67 at 3-shot; greedy span answers 0.57 / 0.43). Llama wiki α35 is the clearest exception at 0-shot (second-largest span shift among Llama models, smallest Llama gain).
- Greedy answers do not contain the gold answer more often (15–19% for every model): what changes is the form of the answer and the probability of the gold span.
| L register α15 | L register α35 | L wiki α15 | L wiki α35 | Q register α15 | Q register α35 | Q wiki α15 | Q wiki α35 | |
|---|---|---|---|---|---|---|---|---|
| benchmark gain, 5 QA tasks (points) | +0.94 | +1.58 | +1.15 | +0.61 | +1.30 | +1.72 | +0.31 | +1.44 |
| Δ P(span), 0-shot (points) | +2.8 | +8.8 | +1.7 | +5.3 | +3.9 | +7.6 | +0.5 | +6.9 |
Sources: preference.csv, preference_vs_base.csv, bench_answer.csv; write-up FINDINGS_answer_content.md.
7 · Steering vectors: more sources, new contrasts
Same contrast, more sources. 16 human sources written before LLM text (Pile: wiki, gutenberg, freelaw, arxiv, pubmed, hackernews, enron, stackexchange; CC-News; Reddit; 6 CORE registers), 4,000 windows of 200–800 tokens each. Each window was paraphrased by SmolLM2-1.7B-Instruct (neutral prompt, best of 10 samples; length ratio 0.85–1.18, ≤ 50% 5-gram overlap, no prompt echo), ≤ 2,000 pairs per source. Vector = whitened mean difference (human − paraphrase) of mean-pooled layer-14 activations, unit norm, the same construction as register. Split-half reliability 0.986–0.997; per-source cosine with register 0.45 (gutenberg) to 0.88 (wiki). register and wiki are the existing vectors from the earlier pipeline (used for all trained models); the composites average the new per-source vectors (equal weight per source):
| vector | human sources (registers) |
|---|---|
register (existing) | 5 registers, 500 samples each: FineWeb, and Pile (monology) PubMed, HackerNews, StackExchange, Wikipedia |
wiki (existing) | Pile (monology) Wikipedia |
all_mean_unit | all 16: Wikipedia, Gutenberg books, FreeLaw court opinions, arXiv papers, PubMed abstracts, HackerNews comments, Enron e-mails, StackExchange Q&A, CC-News articles, Reddit posts, CORE web registers (narrative, opinion, informational, interactive discussion, how-to, informational persuasion) |
no_outliers | the same without Gutenberg, FreeLaw and StackExchange (13) |
expository | Wikipedia, CC-News, arXiv, PubMed, CORE informational |
New contrasts, each aimed at one candidate ingredient (same construction; output-based contrasts are encoded as the steered model sees them: FAQ prompt + source + rewrite, pooling only the rewrite/answer):
| contrast | toward (+) | away from (−) | activations | reliability | held-out sep. | cos register |
|---|---|---|---|---|---|---|
| edu_quality | FineWeb windows, FineWeb-Edu score ≥ 3 | score < 1, length-matched | plain text | 0.97 | 1.00 | -0.01 |
| formulaicity | least predictable of 8 unsteered rewrites (Qwen + Pythia NLL) | most predictable rewrite, same document | generation context | 0.97 | 0.94 | -0.13 |
| temperature | T 1.08 corpus rewrite | base rewrite, same document | generation context | 0.85 | 0.71 | -0.13 |
| answer_style | short direct answer that opens with a source span | answer restating the question, same rewrite | first sentence of the answer | 0.96 | 0.92 | -0.00 |
| faithfulness | answer whose numbers are all in the source | answer with an unsupported number, same rewrite | generation context | 0.94 | 0.80 | -0.04 |
| realized_reg35 | register α35 corpus rewrite | base rewrite, same document | diagnostic: what steering actually changed in the outputs | 0.98 | 0.96 | +0.32 |
Held-out separation = pair accuracy (paired) or AUROC (edu) on a held-out half. reg_plus_faith = register + 0.5 × faithfulness (orthogonalized), α chosen to keep register's push at 35. Random controls: Gaussian directions at the same norm.
Cosines (layer 14)
| vector | register | wiki | all_mean_unit | formulaicity | temperature | faithfulness |
|---|---|---|---|---|---|---|
| all_mean_unit | 0.91 | 0.67 | 1.00 | -0.17 | -0.21 | -0.08 |
| no_outliers | 0.91 | 0.68 | 1.00 | -0.17 | -0.20 | -0.07 |
| expository | 0.92 | 0.74 | 0.95 | -0.17 | -0.20 | -0.08 |
| edu_quality | -0.01 | 0.10 | -0.02 | 0.01 | 0.00 | 0.01 |
| formulaicity | -0.13 | -0.07 | -0.17 | 1.00 | 0.53 | 0.09 |
| temperature | -0.13 | -0.08 | -0.21 | 0.53 | 1.00 | -0.43 |
| answer_style | -0.00 | 0.01 | 0.04 | -0.15 | -0.14 | 0.16 |
| faithfulness | -0.04 | -0.05 | -0.08 | 0.09 | -0.43 | 1.00 |
| realized_reg35 | 0.32 | 0.26 | 0.32 | -0.09 | 0.36 | -0.54 |
| reg_plus_faith | 0.89 | 0.71 | 0.79 | -0.08 | -0.31 | 0.41 |
| random_iso_s0 | 0.02 | -0.00 | 0.03 | 0.05 | 0.03 | -0.01 |
- The human-vs-paraphrase space is almost one-dimensional: every composite has 0.91–0.92 with register.
- All new contrasts are new directions (|cos| ≤ 0.13 with register). Formulaicity is not register (−0.13) but shares a direction with temperature (+0.53).
realized_reg35(register α35 rewrites − base rewrites of the same documents, i.e. what steering actually changed in the outputs) has only 0.32 with register itself and -0.54 with faithfulness.
Screen: what each vector does to the training text
The first 12,000 FineWeb documents of shard 0 rewritten with each vector (production FAQ setup, layer 14), compared document by document with the base corpus. Existing corpora and regenerated arms (*_regen) give the scale.
| set (vs base corpus) | Qwen NLL | Pythia NLL | tokens | capped % | FAQ docs % | short open % | restates Q % | source-span lead % | unsupported nums % | distinct bigrams | edu score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| corpus: reg35 | +0.23 | +0.16 | +134 | +7.6 | -7.8 | +12.6 | -5.2 | -14.8 | +20.6 | -0.008 | -0.05 |
| corpus: wiki35 | +0.18 | +0.12 | +105 | +6.5 | -8.6 | +10.8 | -6.2 | -15.6 | +22.9 | +0.003 | -0.05 |
| corpus: temp108 | +0.22 | +0.20 | +1ns | +0.2ns | -0.7ns | +0.5ns | -5.8 | -4.1 | +2.1 | +0.022 | -0.01 |
| base_regen | -0.02 | -0.01ns | +2ns | -0.1ns | -0.2ns | -0.3ns | +0.2ns | -0.4ns | -1.2ns | +0.000ns | +0.00ns |
| reg35_regen (α35) | +0.23 | +0.16 | +133 | +7.3 | -8.2 | +12.6 | -5.2 | -14.5 | +20.9 | -0.009 | -0.04 |
| all_mean_unit (α35) | +0.20 | +0.14 | +200 | +10.2 | -4.5 | +13.1 | -4.0 | -18.0 | +25.2 | -0.011 | -0.04 |
| no_outliers (α35) | +0.19 | +0.13 | +208 | +10.8 | -5.7 | +12.9 | -3.4 | -18.5 | +24.4 | -0.014 | -0.05 |
| expository (α35) | +0.16 | +0.10 | +214 | +10.1 | -3.9 | +12.2 | -2.7 | -17.6 | +24.7 | -0.016 | -0.04 |
| random_iso_s0 (α35) | -0.03 | -0.03 | +7 | -0.1ns | +2.1 | -1.5 | +4.0 | -6.1 | -0.9ns | -0.006 | +0.05 |
| random_whitened_s0 (α35) | -0.01ns | -0.01ns | +20 | +0.1ns | +2.7 | -1.9 | +3.2 | -5.7 | -0.8ns | -0.007 | +0.04 |
| reg_plus_faith (α39.1) | +0.20 | +0.13 | -44 | +1.4 | -12.4 | +12.1 | -3.8 | +3.2 | -0.6ns | -0.020 | -0.10 |
| formulaicity (α15.3) | +0.40 | +0.36 | -58 | -0.2 | -1.7 | +2.5 | -17.5 | -4.1 | -4.4 | +0.043 | -0.04 |
| formulaicity_a11 (α11) | +0.30 | +0.26 | -45 | -0.3 | -0.7ns | +1.2 | -13.0 | -2.5 | -2.6 | +0.033 | -0.02 |
| formulaicity_a8 (α8) | +0.21 | +0.19 | -32 | -0.0ns | -1.0ns | +0.8 | -9.8 | -2.1 | -2.4 | +0.026 | -0.01ns |
| edu_quality (α34.4) | +0.18 | +0.15 | -47 | -0.2ns | -6.2 | +1.3 | -4.3 | -12.3 | +5.0 | +0.031 | +0.40 |
| edu_quality_a25 (α25) | +0.08 | +0.07 | -31 | -0.3 | -2.7 | +0.5ns | -2.1 | -7.5 | +1.9 | +0.021 | +0.30 |
| edu_quality_a45 (α45) | +0.30 | +0.25 | -63 | -0.2ns | -10.2 | +2.8 | -7.8 | -18.9 | +11.2 | +0.041 | +0.49 |
| temperature (α16.1) | +0.59 | +0.51 | +34 | +1.4 | -1.2 | +5.2 | -19.7 | -17.7 | +9.6 | +0.053 | -0.01ns |
| answer_style (α16.2) | +0.08 | +0.06 | -17 | +0.0ns | -2.0 | +4.3 | -5.5 | +2.5 | -4.4 | +0.003 | -0.04 |
Change vs base corpus; NLL in nats/token under outside scorers (higher = less predictable); percentages in points. ns = 95% paired bootstrap CI includes 0. Base corpus values (contrasts/screen/SCREEN.md): Qwen NLL 1.96, 452 tokens, 74% FAQ docs, 20% unsupported answer numbers, edu 1.52.
How α was chosen: same-contrast composites and random controls use α35, like register (all vectors are unit norm). New contrasts use α = 35 × (spread of activations along the new direction ÷ spread along register), measured on SmolLM2 paraphrase activations at layer 14, so each pushes as far as register relative to natural variation. It's a rough yardstick: formulaicity at α15.3 shifts NLL almost twice as much as register. reg_plus_faith uses α39.1, which keeps its push along register at 35.
α sweep (formulaicity α8/α11, edu α25/α45): formulaicity's NLL shift grows evenly with α and matches register's at ≈ α9, keeping its benefits; edu's score gain saturates (+0.30 / +0.41 / +0.49 at α25 / 34 / 45) while its costs keep growing. Chosen for full arms: formulaicity α9 (effect-matched to register), edu_quality α35, reg_plus_faith α39.
- The screen reproduces the corpora:
reg35_regenmatches the register α35 corpus on every measure. - The direction matters: random directions at the same norm barely change the text (-0.03 NLL vs +0.23).
- More sources of the same contrast = register with more cost: all_mean_unit / no_outliers / expository give register's profile with longer outputs (+200 to +214 vs +134 tokens), more cap hits and more unsupported numbers.
- reg_plus_faith keeps register's NLL shift and short openings but brings unsupported numbers back to base (-0.6 vs +20.6 points) and removes the length inflation; price: -12.4 points of FAQ-shaped documents.
- formulaicity gives the largest NLL shift with fewer unsupported numbers than base and much less question-restating, but almost no short-opening effect: it separates "diversity" from register's answer style.
- edu_quality raises the outputs' edu score (+0.40; register lowers it). temperature overshoots and costs faithfulness; answer_style is weak at its α.
Sources: vector_sources.csv, vector_cosines.csv, contrast_reports.csv, screen.csv; write-ups vector_computation/results/COMPARISON.md, contrasts/README.md.
8 · New arms at full scale: every steered corpus beats base at both sizes; reg_plus_faith leads
The three vectors chosen in the screen (§7), and two that add two of them (rpf_plus_edu100 = register + 0.5 · faithfulness + 1.0 · edu_quality, and rpf_plus_edu50 with edu weight 0.5; the two parts are orthogonal, cosine −0.008), were used to generate full corpora (10B synthetic tokens each) and to pretrain Qwen2 1.7B and Qwen2 2.9B from scratch, with the same mix, training setup and evaluation as §1–2. One seed per model.
- All 12 steered runs improve the macro score (6 corpora × 2 model sizes), so the effect is not tied to one vector or one model size.
- reg_plus_faith α39 gives the largest gain we have measured: +1.42 [+1.34, +1.51] at 1.7B and +0.98 [+0.89, +1.06] at 2.9B, above the seed-noise band at both sizes and ahead of register α35 in both (+0.40 [+0.32, +0.48] and +0.27 [+0.19, +0.35]). The faithfulness correction does not cost the benchmark gain: it adds to it.
- The combined vectors come close: with edu weight 1.0, +1.39 at 1.7B and +0.73 at 2.9B; with edu weight 0.5, +1.33 and +0.89. Both add the knowledge gains that reg_plus_faith lacks (below), and the lighter weight holds up better with scale.
Three measures
All tasks are run in cloze form (3-shot, no answer options in the prompt). The cloze score is the probability the model gives the correct answer; all 15 tasks have it and the macro score is built from it. For the 8 multiple-choice tasks, accuracy asks whether the correct option scores higher than the wrong ones. PMI-accuracy is the same after subtracting each option's score without the question. We include it because it removes the advantage of options that are likely whatever the question is, so it shows whether the question itself points the model to the right answer; a gain seen on both accuracy measures is more credible than one seen on either alone. It exists for the 5 tasks whose FinePhrase definition includes it.
Cloze score, per task
| WikiTableQ | TREB-QA | DROP | SQuADv2 | TriviaQA | XCSQA | ARC-Easy | ARC-Challenge | OpenBookQA | HellaSwag | PIQA | WinoGrande | MMLU-Redux | GSM8K | LAMBADA | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Q1.7B · reg_plus_faith α39 | +4.00† | +2.79† | +2.58† | +6.47† | -0.44† | +0.41† | +0.02 | +0.43† | +0.28† | +0.15† | -0.08† | +0.03 | +0.09 | +0.28† | +0.48† |
| Q1.7B · rpf_plus_edu50 α39 | +2.52† | +2.19† | +2.41† | +5.51† | +0.07 | +0.43† | +1.05† | +0.74† | +0.34† | +0.23† | +0.13† | +0.19† | +0.36† | +0.44† | +1.62† |
| Q1.7B · rpf_plus_edu100 α39 | +1.37† | +2.06† | +2.15† | +5.69† | -0.04 | +0.83† | +2.03† | +1.13† | +0.59† | +0.21† | +0.13† | +0.34† | +0.46† | +0.67† | +0.92† |
| Q1.7B · register α35 | +4.32† | +1.65† | +2.75† | -0.10 | +0.09 | +0.60† | +0.42† | +0.37† | +0.53† | +0.15† | +0.00 | +0.34† | +0.28† | +0.87† | -0.86† |
| Q1.7B · edu_quality α35 | +0.58† | +0.95† | +0.94† | +0.57† | -0.33† | +0.82† | +2.65† | +1.57† | +0.73† | +0.05† | +0.16† | +0.24† | +0.27† | -0.08 | +0.02 |
| Q1.7B · formulaicity α9 | +0.83† | +0.45 | +0.12 | +2.44† | -0.56† | -0.41† | +0.26 | +0.40† | +0.59† | +0.11† | -0.02 | +0.16† | -0.08 | +0.98† | +0.13 |
| Q2.9B · reg_plus_faith α39 | +2.42† | +1.02† | +3.99† | +3.28† | +0.14† | +0.86† | -0.44† | -0.05 | +0.22† | +0.02 | -0.03 | -0.07 | +0.04 | +0.43† | -0.18 |
| Q2.9B · rpf_plus_edu50 α39 | +1.89† | +0.37 | +3.99† | +2.97† | +0.59† | +1.36† | +0.95† | +0.24† | +0.24† | +0.08† | +0.16† | -0.04 | +0.30† | -0.49† | +0.95† |
| Q2.9B · rpf_plus_edu100 α39 | -0.26† | -0.09 | +1.28† | +3.01† | +0.55† | +1.21† | +1.77† | +0.62† | +0.31† | +0.15† | +0.18† | -0.18† | +0.35† | +0.82† | +0.93† |
| Q2.9B · register α35 | +1.70† | +0.13 | +4.99† | -1.31† | +0.43† | +1.41† | -0.07 | +0.11 | +0.06 | -0.05† | -0.00 | -0.43† | +0.10 | +1.25† | -1.06† |
| Q2.9B · edu_quality α35 | -0.53† | -0.12 | +2.50† | +0.70† | +0.04 | +1.07† | +2.80† | +1.28† | +0.45† | -0.00 | +0.29† | -0.25† | +0.19† | +0.74† | +0.09 |
| Q2.9B · formulaicity α9 | -0.45† | +0.45 | +1.57† | +0.36 | +0.12† | +0.60† | -0.13 | +0.04 | +0.19 | -0.00 | +0.12† | -0.13† | +0.16† | +0.41† | -0.17 |
Cloze score: Δ prob_norm_token × 100 vs base, full evaluation sets; † = BH q < 0.05. Register α35 rows repeat §1 for reference.
- reg_plus_faith turns register's SQuADv2 cost into its largest gain (+6.47 / +3.28 vs -0.10 / -1.31 for register) while keeping most of register's DROP and table gains. Its training answers lead with a source span more often than base (§7), and SQuADv2 rewards exactly that.
- edu_quality adds what register lacks: knowledge. ARC-Easy +2.65 / +2.80, the most stable effect across sizes.
- The combined vector inherits both: SQuADv2 +5.69 / +3.01 and ARC-Easy +2.03 / +1.77, and the two combined vectors are the only arms with a clear LAMBADA gain at both sizes (+0.92 / +0.93; edu weight 0.5: +1.62 / +0.95). What the 1.0 weight gives up is part of the table and DROP gain (WikiTableQ +1.37 / -0.26).
Accuracy and PMI-accuracy
| XCSQA | HellaSwag | ARC-Easy | WinoGrande | PIQA | ARC-Challenge | OpenBookQA | MMLU-Redux | mean | |
|---|---|---|---|---|---|---|---|---|---|
| Q1.7B · reg_plus_faith α39 | +1.90 | +0.22 | +0.93 | +0.79 | -0.33 | +0.60 | +0.00 | +0.15 | +0.53 |
| Q1.7B · rpf_plus_edu50 α39 | +2.00 | +1.60† | +2.19† | +0.47 | -0.16 | -0.09 | +0.60 | +0.31 | +0.87 |
| Q1.7B · rpf_plus_edu100 α39 | +2.90† | +1.29† | +2.69† | +0.63 | +0.82 | +1.71 | +0.00 | +0.64 | +1.34 |
| Q1.7B · register α35 | +2.80 | +1.49† | +0.76 | +0.95 | +0.27 | +1.71 | +2.60 | -0.07 | +1.31 |
| Q1.7B · edu_quality α35 | +3.20† | +1.26† | +3.66† | +0.16 | -0.82 | +3.58† | +1.60 | +0.42 | +1.63 |
| Q1.7B · formulaicity α9 | -0.80 | +0.72 | +0.25 | -0.16 | +0.16 | +1.37 | -1.80 | +0.15 | -0.01 |
| Q2.9B · reg_plus_faith α39 | +2.10 | +0.68 | +1.26 | -0.24 | +0.49 | +0.26 | +2.00 | -0.10 | +0.81 |
| Q2.9B · rpf_plus_edu50 α39 | +1.40 | +1.23† | +3.54† | -0.95 | +1.20 | +0.34 | +0.60 | +0.25 | +0.95 |
| Q2.9B · rpf_plus_edu100 α39 | +1.40 | +1.62† | +3.66† | -0.63 | +0.54 | -0.26 | +1.80 | +0.20 | +1.04 |
| Q2.9B · register α35 | +4.40† | +1.00† | +3.11† | -1.03 | +0.82 | -0.43 | +0.00 | -0.98 | +0.86 |
| Q2.9B · edu_quality α35 | +2.50 | +1.00† | +4.67† | -0.95 | +0.16 | -0.51 | +0.80 | -0.46 | +0.90 |
| Q2.9B · formulaicity α9 | +1.80 | +0.81† | +2.27† | -2.68 | +0.92 | -0.68 | -1.20 | -0.50 | +0.09 |
Accuracy: Δ acc_norm_token vs base, percentage points; † = BH q < 0.05. Mean = unweighted mean of the 8 task deltas.
| XCSQA | PIQA | WinoGrande | OpenBookQA | MMLU-Redux | mean | |
|---|---|---|---|---|---|---|
| Q1.7B · reg_plus_faith α39 | +3.40 | -0.49 | +2.05 | +2.40 | +0.43 | +1.56 |
| Q1.7B · rpf_plus_edu50 α39 | +4.30† | +0.44 | +0.39 | +3.60 | -0.39 | +1.67 |
| Q1.7B · rpf_plus_edu100 α39 | +3.20 | +2.45 | +1.42 | +4.60 | +0.52 | +2.44 |
| Q1.7B · register α35 | +4.40† | +1.41 | +1.18 | +4.80 | +0.09 | +2.38 |
| Q1.7B · edu_quality α35 | +2.00 | +0.00 | +1.58 | +2.80 | +0.79 | +1.43 |
| Q1.7B · formulaicity α9 | +0.20 | -0.60 | +0.63 | +1.60 | -0.66 | +0.23 |
| Q2.9B · reg_plus_faith α39 | +5.20† | +1.25 | +0.32 | +1.60 | -0.35 | +1.60 |
| Q2.9B · rpf_plus_edu50 α39 | +4.30† | +1.80 | -1.18 | +1.80 | -0.06 | +1.33 |
| Q2.9B · rpf_plus_edu100 α39 | +4.30† | +2.07 | +0.08 | +8.60† | +0.48 | +3.11 |
| Q2.9B · register α35 | +5.40† | +3.21† | -1.66 | +3.60 | -0.31 | +2.05 |
| Q2.9B · edu_quality α35 | +2.40 | +1.69 | +0.47 | +4.20 | +0.15 | +1.78 |
| Q2.9B · formulaicity α9 | +2.30 | +1.03 | -1.03 | +3.20 | +0.29 | +1.16 |
PMI-accuracy: Δ acc_norm_pmi vs base, percentage points, on the 5 tasks whose definition includes it; † = BH q < 0.05.
- The gains are not only higher probability on the answer: the models choose better. Mean accuracy rises in 11 of 12 runs and mean PMI-accuracy in 12 of 12.
- The combined vector with edu weight 1.0 is the strongest arm here: the largest PMI-accuracy gain at both sizes (+2.44 / +3.11) and the largest accuracy gain at 2.9B (+1.04; at 1.7B edu_quality leads with +1.63). ARC-Easy accuracy is its clearest single gain (+2.7 / +3.7).
- Single-task accuracy changes are noisy at these test sizes (500–10,000 questions); the means and ARC-Easy are the reliable parts.
Held-out loss
| FineWeb | Wikipedia | arXiv | HackerNews | StackExchange | FreeLaw | Gutenberg | Mean | |
|---|---|---|---|---|---|---|---|---|
| Q1.7B · reg_plus_faith α39 | +1.83† | +1.43† | +3.07† | +3.37† | +2.46† | +1.93† | +2.27† | +2.34† |
| Q1.7B · rpf_plus_edu50 α39 | +1.91† | +2.11† | +1.77† | +3.56† | +1.06† | +1.48† | +2.34† | +2.03† |
| Q1.7B · rpf_plus_edu100 α39 | +1.74† | +1.75† | +0.58† | +2.54† | -1.11† | +0.69† | +1.80† | +1.14† |
| Q1.7B · register α35 | -0.44† | +1.03† | +2.17† | +2.02† | +2.18† | +0.18† | +0.33† | +1.07† |
| Q1.7B · edu_quality α35 | +1.13† | +0.66† | -2.98† | +0.68† | -4.15† | -1.47† | +0.38† | -0.82† |
| Q1.7B · formulaicity α9 | +1.20† | +0.71† | +0.22† | +2.06† | +1.89† | -0.53† | +0.43† | +0.85† |
| Q2.9B · reg_plus_faith α39 | +2.02† | +1.80† | +3.58† | +3.93† | +2.94† | +2.20† | +2.56† | +2.72† |
| Q2.9B · rpf_plus_edu50 α39 | +2.08† | +2.40† | +2.46† | +3.84† | +1.28† | +1.94† | +2.83† | +2.41† |
| Q2.9B · rpf_plus_edu100 α39 | +2.01† | +2.20† | +1.65† | +3.31† | -0.57† | +1.33† | +2.48† | +1.78† |
| Q2.9B · register α35 | -0.77† | +0.75† | +2.09† | +2.28† | +2.36† | -0.00 | -0.01 | +0.96† |
| Q2.9B · edu_quality α35 | +1.56† | +1.46† | -1.78† | +2.16† | -3.29† | -0.31† | +1.29† | +0.16† |
| Q2.9B · formulaicity α9 | +1.43† | +1.19† | +0.03 | +2.02† | +2.20† | +0.13† | +0.90† | +1.13† |
Held-out loss gain vs base in centi-nats on the full splits (positive = lower loss); † = 95% paired bootstrap CI excludes 0.
- reg_plus_faith improves every domain at both sizes (7 of 7 at 1.7B, 7 of 7 at 2.9B; mean +2.34 / +2.72 centi-nats, more than twice register's +1.07 / +0.96 and above the best earlier arm, wiki α15 at +1.91 on Qwen 1.7B), including FineWeb, where register α35 is worse than base. It leads on both benchmarks and loss, so the trade-off of §2 is not inherent to steering.
- The combined vector improves 6 / 6 of 7 domains (mean +1.14 / +1.78) and is the best arm on Wikipedia; like edu_quality it pays on arXiv and StackExchange.
Effect of model size
| arm (gain vs base: 1.7B → 2.9B) | macro score | cloze, 8 multiple-choice | cloze, 7 without options | accuracy (8) | PMI-accuracy (5) | held-out loss |
|---|---|---|---|---|---|---|
| reg_plus_faith α39 | +1.42 → +0.98 | +0.17 → +0.07 | +2.31 → +1.59 | +0.53 → +0.81 | +1.56 → +1.60 | +2.34 → +2.72 |
| rpf_plus_edu50 α39 | +1.33 → +0.89 | +0.43 → +0.41 | +2.11 → +1.47 | +0.87 → +0.95 | +1.67 → +1.33 | +2.03 → +2.41 |
| rpf_plus_edu100 α39 | +1.39 → +0.73 | +0.72 → +0.55 | +1.83 → +0.89 | +1.34 → +1.04 | +2.44 → +3.11 | +1.14 → +1.78 |
| register α35 | +1.03 → +0.71 | +0.34 → +0.14 | +1.25 → +0.88 | +1.31 → +0.86 | +2.38 → +2.05 | +1.07 → +0.96 |
| edu_quality α35 | +0.60 → +0.67 | +0.81 → +0.73 | +0.38 → +0.49 | +1.63 → +0.90 | +1.43 → +1.78 | -0.82 → +0.16 |
| formulaicity α9 | +0.53 → +0.27 | +0.13 → +0.11 | +0.62 → +0.33 | -0.01 → +0.09 | +0.23 → +1.16 | +0.85 → +1.13 |
Each cell: mean gain over base at Qwen2 1.7B → at Qwen2 2.9B. Cloze and macro in points, accuracy in percentage points, loss in centi-nats.
| arm (cloze gain vs base: 1.7B → 2.9B) | general knowledge | reading comprehension | tables | reasoning | language understanding | math |
|---|---|---|---|---|---|---|
| reg_plus_faith α39 | +0.05 → -0.20 | +4.53 → +3.64 | +3.39 → +1.72 | +0.20 → +0.35 | +0.09 → -0.03 | +0.28 → +0.43 |
| rpf_plus_edu50 α39 | +0.70 → +0.63 | +3.96 → +3.48 | +2.36 → +1.13 | +0.30 → +0.59 | +0.21 → +0.02 | +0.44 → -0.49 |
| rpf_plus_edu100 α39 | +1.25 → +1.06 | +3.92 → +2.14 | +1.71 → -0.18 | +0.52 → +0.56 | +0.27 → -0.01 | +0.67 → +0.82 |
| register α35 | +0.35 → +0.01 | +1.33 → +1.84 | +2.98 → +0.92 | +0.38 → +0.49 | +0.25 → -0.24 | +0.87 → +1.25 |
| edu_quality α35 | +1.46 → +1.50 | +0.75 → +1.60 | +0.77 → -0.32 | +0.57 → +0.60 | +0.15 → -0.12 | -0.08 → +0.74 |
| formulaicity α9 | +0.09 → +0.01 | +1.28 → +0.97 | +0.64 → +0.00 | +0.05 → +0.30 | +0.13 → -0.07 | +0.98 → +0.41 |
Cloze gain by task category (mean of the per-task gains in the category; no intervals).
- The gains carry over to the larger model. All arms stay above base on the macro score, and reg_plus_faith stays first at both sizes.
- Cloze gains shrink by a third to a half (edu_quality holds). The reduction is on SQuADv2, tables and TREB-QA, where the 2.9B base model improves most on its own, so there is less left to add.
- Accuracy converges and PMI-accuracy holds or grows. At 2.9B every arm except formulaicity gains +0.81 to +1.04 accuracy points, and mean PMI-accuracy gains are as large or larger than at 1.7B for four of the six arms. The part of the effect that reflects choosing the right answer does not fade with scale.
- Held-out loss gains hold or grow for every arm except register (a small drop).
- By category, the education component is what keeps knowledge gains at scale: general-knowledge gains vanish at 2.9B for register and reg_plus_faith (+0.01, -0.20) but stay for the arms with an education part (+0.63 to +1.06). For the four register-based arms, table gains shrink by about half or more, reasoning gains hold or grow, and language-understanding gains fade.
How the combined vector was chosen
| set (vs base corpus) | Qwen NLL | tokens | FAQ docs % | short open % | source-span lead % | unsupported nums % | edu score |
|---|---|---|---|---|---|---|---|
| reg_plus_faith α39 | +0.20 | -44 | -12.4 | +12.1 | +3.2 | -0.6ns | -0.10 |
| edu_quality α34 (trained at α35) | +0.18 | -47 | -6.2 | +1.3 | -12.3 | +5.0 | +0.40 |
| rpf_plus_edu100 α39 (trained) | +0.20 | -79 | -12.3 | +10.1 | -5.7 | +1.9 | +0.18 |
| + 0.5 · edu, α43 | +0.22 | -75 | -14.3 | +12.7 | -1.8 | -0.7ns | +0.06 |
| + 0.7 · edu, α46 | +0.24 | -84 | -15.1 | +13.1 | -4.9 | +1.5ns | +0.12 |
| + 1.0 · edu, α53 | +0.28 | -91 | -17.9 | +13.8 | -9.9 | +4.4 | +0.22 |
Change vs base corpus, as in §7. "+ w · edu" rows keep reg_plus_faith's push along register (35) and add edu on top; the trained arm keeps reg_plus_faith's total strength (α39) instead.
- At α39 the edu score rises (+0.18 vs -0.10 for reg_plus_faith) while unsupported numbers stay at base level (+1.9 points vs +20.6 for register). Source-span openings fall part of the way back (-5.7 points), which fits the smaller table and DROP gains after training.
- The trained models moved the way the screen suggested (more knowledge, fewer span-led answers), which supports using the screen to pick vectors. A smaller edu weight was the natural next setting (next).
A lighter education weight: rpf_plus_edu50
| set at α39 (vs base corpus) | Qwen NLL | tokens | FAQ docs % | short open % | source-span lead % | unsupported nums % | edu score |
|---|---|---|---|---|---|---|---|
| edu weight 0 (reg_plus_faith α39, trained) | +0.20 | -44 | -12.4 | +12.1 | +3.2 | -0.6ns | -0.10 |
| edu weight 0.5 (trained) | +0.20 | -73 | -13.5 | +11.1 | -1.2 | -0.1ns | +0.05 |
| edu weight 0.6 | +0.21 | -73 | -12.9 | +10.5 | -2.3 | -0.5ns | +0.09 |
| edu weight 0.7 | +0.21 | -76 | -12.7 | +11.1 | -3.4 | -1.1ns | +0.11 |
| edu weight 1.0 (trained) | +0.20 | -79 | -12.3 | +10.1 | -5.7 | +1.9 | +0.18 |
All sets at the same total strength (α39). The weight sets how the push is shared: at 1.0 register and edu get 26 each, at 0.5 register gets 32 and edu 16.
- The screen showed a smooth trade: as the edu weight goes from 0.5 to 1.0 the edu score rises (+0.05 to +0.18) and source-span openings fall (-1.2 to -5.7 points). We trained 0.5.
- After training it lands between its parents, as intended. It keeps about half of the 1.0 weight's ARC-Easy gain (+1.05 / +0.95), nearly all of the SQuADv2 gain (+5.51 / +2.97), and wins back much of the table and DROP gains (WikiTableQ +2.52 / +1.89; DROP +2.41 / +3.99).
- It is the more robust combined vector: it keeps 67% of its macro gain at 2.9B (edu weight 1.0: 53%), is ahead of the 1.0 weight there (+0.16 [+0.08, +0.24]), and improves held-out loss in all 7 domains at both sizes (+2.03 / +2.41 centi-nats).
- Against reg_plus_faith it is -0.10 [-0.17, -0.02] at 1.7B and -0.09 [-0.16, -0.02] at 2.9B on the macro score: a small gap. Its one weak spot is GSM8K cloze at 2.9B (-0.49).
Sources: aggregates, task deltas (all three measures), held-out loss (and the qwen29 files), screen_rpf_edu.csv, screen_rpf_edu_weights.csv; computed by model_results_analysis/compare_arms.py and contrasts/screen/screen_compare.py.
9 · Faster corpus generation
Generation (vLLM, SmolLM2-1.7B on GH200) now runs 3 engines per GPU (memory share 0.30 each, 768 concurrent sequences, suffix speculative decoding) instead of one large engine. A sweep of serving settings found this the fastest: 1.19× (steered) and 1.24× (unsteered) more output per GPU. A 120k-document test reproduced it: 7,750 vs 6,681 output tokens/s per GPU (1.16×), with the same output distribution (579 vs 579 tokens per document, 8.5% vs 8.6% hitting the length cap; every metric of the distribution check within 3–11% of the steering effect). Each shard takes ~2.5× longer because three share a GPU, so jobs are sized at 36 shards per 3 nodes (one round, ~10–13 h). About a sixth fewer GPU-hours per corpus.
Sources: optimized_inference/README.md, throughput_sweep/results/SUMMARY.md, throughput_sweep/VALIDITY.md.
10 · What is shown and what is not
| statement | status | evidence |
|---|---|---|
| Steered models improve held-out loss; wiki more than register; α35 less than α15 | shown | §2 |
| Loss gains sit on hard/rare tokens and whitespace; α35 loses function words and prose words | shown | §3 |
| Benchmark gains are on the gold answer, not on reading the context | shown | §4 |
| At α35, training answers open more directly and contain more added numbers | shown | §5 |
| Register and wiki training data are nearly identical | shown | §3, §5 |
| Steered models prefer span answers (0-shot and 3-shot) | shown | §6 |
| Register models prefer spans more than wiki models | only 0-shot | §6 |
| The span preference follows the benchmark gains | partly (ρ 0.43–0.81, exceptions) | §6 |
| Smaller models mostly gain answer probability; 2.9B gains accuracy (better choices) | suggestive (one seed per size) | §1 |
| The steering effect on the training text is specific to the direction (random ≈ base) | shown (corpus level) | §7 |
| A faithfulness correction removes the unsupported numbers and keeps the NLL shift | shown (corpus level) | §7 |
| New contrasts (reg_plus_faith, edu_quality, formulaicity, combined) improve benchmarks at Qwen 1.7B and 2.9B | shown (12 of 12 runs; one seed per size) | §8 |
| reg_plus_faith improves held-out loss in all 7 domains at both sizes, with no FineWeb cost | shown | §8 |
| reg_plus_faith beats register α35 on benchmarks | consistent at both sizes; each gap within seed noise | §8 |
| edu_quality improves science-knowledge tasks (ARC) at both sizes | shown (probability; accuracy on ARC-Easy at both sizes) | §8 |
| A vector combining reg_plus_faith and edu_quality keeps the SQuADv2 gain and the ARC gain after training | shown (one seed per size) | §8 |
| The combined vector gives the largest accuracy / PMI-accuracy gains | PMI-accuracy at both sizes, accuracy at 2.9B; margins within item noise | §8 |
| The combined vector beats reg_plus_faith on the macro score | no: level at 1.7B, behind at 2.9B (smaller table and DROP gains) | §8 |
| Halving the education weight keeps the SQuADv2 gain, about half the ARC gain, and recovers table, DROP and loss gains | shown (one seed per size) | §8 |
| rpf_plus_edu50 beats reg_plus_faith on the macro score | no: about 0.1 points behind at both sizes; ahead on accuracy and general knowledge | §8, §11 |
| Steering gains persist at the larger model | shown: cloze gains a third to a half smaller, accuracy-type and loss gains stable | §8 |
| Register really beats wiki on benchmarks | not established (within seed noise) | §1 |
| The changed training answers cause the span preference, or the preference causes the gains | not shown (correlational, one seed) | — |
Caveats
- One training seed per model. CIs on this page cover evaluation/item noise only.
- The new arms (§8) also use different human shards than base and register (their synthetic documents are shorter), the same confound as below.
- α35 mixes also contain different human text (other CommonCrawl dumps, because longer synthetic documents shift which FineWeb shards fill the human half); this is collinear with α. Its measured effect on predictability is ~10× smaller than the synthetic difference, but it is not ruled out.
- Source-document faithfulness concerns a document the model never sees during training.
Next
- Seeds (Qwen base, reg_plus_faith α39, rpf_plus_edu50 α39, register α35): every comparison above is judged against a seed SD we haven't measured; a second seed of the two leading vectors at 1.7B would settle their order (§11).
- Mechanism of reg_plus_faith: split SQuADv2 / DROP items into copied-span vs computed answers to test the "more extractive training answers" explanation.
- Temperature-matched baseline (T 1.08 corpus, generated): does less predictable text alone give the gains?
11 · Which steering vector is best so far
reg_plus_faith α39 first, rpf_plus_edu50 α39 a close second. Criteria, in order of priority: cloze score, accuracy, PMI-accuracy, held-out loss; and a preference for vectors that improve some areas without making others worse.
| criterion (gain vs base, 1.7B / 2.9B) | reg_plus_faith α39 | rpf_plus_edu50 α39 |
|---|---|---|
| 1 · cloze macro score | +1.42 / +0.98 | +1.33 / +0.89 |
| 2 · mean accuracy (8 tasks) | +0.53 / +0.81 | +0.87 / +0.95 |
| 3 · mean PMI-accuracy (5 tasks) | +1.56 / +1.60 | +1.67 / +1.33 |
| 4 · held-out loss gain (centi-nats) | +2.34 / +2.72 | +2.03 / +2.41 |
| cloze tasks significantly below base | 1.7B: TriviaQA -0.44, PIQA -0.08; 2.9B: ARC-Easy -0.44 | 1.7B: none; 2.9B: GSM8K -0.49 |
| accuracy / PMI-accuracy tasks significantly below base | 0 | 0 |
| loss domains worse than base (of 14) | 0 | 0 |
| share of cloze gain kept at 2.9B | 69% | 67% |
| general-knowledge cloze gain | +0.05 / -0.20 | +0.70 / +0.63 |
- reg_plus_faith wins the first criterion at both sizes (rpf_plus_edu50 is -0.10 [-0.17, -0.02] and -0.09 [-0.16, -0.02] behind) and has the largest loss gain. It is also the simpler recipe: two components.
- rpf_plus_edu50 is close on cloze, ahead on accuracy at both sizes, and more evenly spread: it keeps a clear general-knowledge gain at both sizes (+0.70 / +0.63; reg_plus_faith +0.05 / -0.20) on top of the reading-comprehension and table gains. It is clearly ahead of register α35 (+0.30 [+0.23, +0.37] / +0.18 [+0.10, +0.26]).
- Both come close to "improvement without a trade-off": neither loses on any accuracy or PMI-accuracy task or on any loss domain; reg_plus_faith has three small cloze drops and rpf_plus_edu50 one (table). For rpf_plus_edu50, GSM8K exact-match is also below base at 1.7B (-0.91 points on a base of 2.27).
- The other arms drop out: register has clear costs (SQuADv2, LAMBADA, FineWeb loss), rpf_plus_edu100 keeps only 53% of its cloze gain at 2.9B and loses the table gain, and edu_quality and formulaicity gain less.
- Caveat: the gap between the two leaders is about 0.1 points on one seed each; the intervals cover question sampling, not training-run variation. That both beat base and register is the safer claim.