Steered paraphrase pretraining: what changes, and where the gains come from
Small LMs pretrained on FAQ paraphrases generated with activation steering (register or wiki direction). Results summary; every number below is computed from the CSVs in data/.
Summary
- Benchmarks: macro-score changes range from +0.01 (not significant, Llama wiki α35) to +1.03 points; only Qwen2 1.7B register α35 is clearly above seed noise. Register leads wiki, by margins within seed noise.
- Language modelling: every steered model lowers held-out loss by ~1–2% perplexity, wiki most and α35 least: the reverse of the benchmark ranking.
- Mechanism (loss): gains on hard/rare tokens and layout; α35 gives up gains on function words and prose.
- Mechanism (benchmarks): gains are on the gold answer, not on reading the context. Steered models prefer short span answers, most at α35, and this roughly tracks the benchmark gains.
- Data: steering makes FAQ answers open more directly and contain more numbers that are not in the source document; register and wiki data are nearly identical. What separates register from wiki is not known.
Setup
- Models: Llama 1B and Qwen2 1.7B, pretrained from scratch, one seed each. Each architecture has a base model and four steered models.
- Data: one epoch over a 20B-token mix, 50:50 synthetic and human (FineWeb). The synthetic half is FineWeb rewritten as FAQs by SmolLM2-1.7B-Instruct. Base: no steering. Register and wiki: a steering vector added at layer 14 with strength α = 15 or 35.
- Units: benchmark Δ in points (prob_norm_token × 100); loss Δ in centi-nats (nats × 100), positive = better than base. CIs are 95% paired bootstraps.
1 · Benchmarks: gains, but mostly within seed noise (1 seed only)
Paper-style evaluation (lighteval, 3-shot, cloze formulation, prob_norm_token), every question of every task, paired bootstrap against each architecture's base model. Bars show the change in the macro score with its 95% CI; the shaded band is the seed-to-seed standard deviation of this score reported by the finephrase paper (0.16–0.49 points). One seed per model.
- The largest gain is Qwen2 1.7B register α35: +1.03 [+0.95, +1.11], the only one clearly above the seed-noise band. Llama wiki α35 shows no gain (+0.01).
- Base macro scores: Llama 16.44, Qwen 17.53.
Direct comparisons (macro Δ, points, 95% CI)
| contrast | Llama 1B | Qwen2 1.7B |
|---|---|---|
| register − wiki, α15 | +0.12 [+0.06, +0.19] | +0.29 [+0.22, +0.36] |
| register − wiki, α35 | +0.38 [+0.31, +0.44] | +0.38 [+0.31, +0.46] |
| α35 − α15, register | -0.06 [-0.13, +0.01] | +0.37 [+0.30, +0.45] |
| α35 − α15, wiki | -0.32 [-0.38, -0.26] | +0.28 [+0.21, +0.35] |
Register beats wiki in all four comparisons, but each margin (≤ 0.38) is within the noise expected between two single-seed models (√2 × seed SD ≈ 0.23–0.69). α35 helps Qwen but not Llama.
Per task
| WikiTableQ | TREB-QA | DROP | SQuADv2 | TriviaQA | XCSQA | ARC-Easy | ARC-Challenge | OpenBookQA | HellaSwag | PIQA | WinoGrande | MMLU-Redux | GSM8K | LAMBADA | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama 1B · register α15 | +0.50† | +0.09 | +2.13† | +1.50† | +0.17† | +0.88† | +0.56† | +0.40† | +0.18† | +0.15† | +0.23† | +0.39† | +0.28† | -0.55† | +0.24 |
| Llama 1B · register α35 | +2.52† | +0.88† | +2.81† | +0.54† | +0.69† | +1.21† | +0.84† | +0.33† | +0.28† | +0.07† | +0.08† | +0.28† | +0.33† | -2.35† | -0.86† |
| Llama 1B · wiki α15 | +0.39† | +0.94† | +1.51† | +1.36† | +0.53† | +0.70† | +0.54† | +0.31† | +0.12† | +0.04† | +0.25† | +0.57† | +0.06 | -1.11† | -0.26 |
| Llama 1B · wiki α35 | +0.84† | -0.16 | +2.26† | -0.94† | +0.79† | +1.28† | +1.04† | +0.35† | +0.20† | +0.09† | +0.34† | +0.42† | +0.35† | -2.51† | +0.25 |
| Qwen2 1.7B · register α15 | +2.35† | +0.96† | +2.18† | +0.51† | +0.27† | +0.10 | +0.33† | +0.23† | +0.20† | +0.18† | +0.09† | -0.04 | +0.14† | +0.47† | +0.05 |
| Qwen2 1.7B · register α35 | +4.32† | +1.65† | +2.75† | -0.10 | +0.09 | +0.60† | +0.42† | +0.37† | +0.53† | +0.15† | +0.00 | +0.34† | +0.28† | +0.87† | -0.86† |
| Qwen2 1.7B · wiki α15 | +1.90† | +0.91† | -0.71† | -0.96† | +0.15† | +0.56† | +0.48† | +0.30† | +0.38† | +0.23† | +0.12† | +0.04 | +0.13 | +0.80† | +0.53† |
| Qwen2 1.7B · wiki α35 | +3.43† | +1.60† | +2.77† | -1.57† | +0.22† | +0.34† | +0.41† | +0.38† | +0.42† | +0.21† | +0.19† | +0.28† | -0.00 | -0.03 | -0.08 |
Δ prob_norm_token × 100 vs base, full evaluation sets. † = significant after BH correction (q < 0.05). Blue = better than base, red = worse. Hover a cell for its value.
- 93 of 120 model × task changes are significantly positive, 10 significantly negative.
- Most consistent gains: WikiTableQ (+0.39 to +4.32) and DROP (+1.51 to +2.81 in 7 of 8 models; Qwen wiki α15 -0.71).
- Consistent costs: LAMBADA for register α35 (Llama -0.86, Qwen -0.86) and GSM8K on Llama (all four models, down to -2.51).
Sources: aggregates, task deltas, contrasts (and the qwen files); write-up LIGHTEVAL_NALL.md.
2 · Held-out loss: every model improves, wiki most, α35 least
Per-domain held-out loss, recomputed on the full evaluation splits (the final_eval.json written at the end of training held the FineWeb loss under every domain name). Paired bootstrap over sequences.
| FineWeb | Wikipedia | arXiv | HackerNews | StackExchange | FreeLaw | Gutenberg | Mean | |
|---|---|---|---|---|---|---|---|---|
| Llama 1B · register α15 | +0.62† | +1.38† | +3.70† | +1.73† | +2.90† | +1.08† | +1.63† | +1.86† |
| Llama 1B · register α35 | -0.29† | +1.27† | +2.75† | +2.52† | +2.54† | +0.20† | +0.34† | +1.33† |
| Llama 1B · wiki α15 | +0.32† | +1.78† | +4.80† | +1.74† | +2.99† | +1.13† | +1.49† | +2.04† |
| Llama 1B · wiki α35 | -0.36† | +2.90† | +5.60† | +1.84† | +2.95† | +0.35† | +0.48† | +1.96† |
| Qwen2 1.7B · register α15 | +0.51† | +1.26† | +3.22† | +1.62† | +2.66† | +1.10† | +1.22† | +1.66† |
| Qwen2 1.7B · register α35 | -0.44† | +1.03† | +2.17† | +2.02† | +2.18† | +0.18† | +0.33† | +1.07† |
| Qwen2 1.7B · wiki α15 | +0.40† | +1.69† | +4.20† | +1.11† | +3.10† | +1.33† | +1.54† | +1.91† |
| Qwen2 1.7B · wiki α35 | -0.40† | +2.50† | +4.80† | +1.11† | +2.83† | +0.52† | +0.25† | +1.66† |
Δ loss vs base in centi-nats (nats × 100) on the full held-out splits; positive = lower loss than base. † = BH q < 0.05.
- Mean gain: wiki α15 +2.04 / +1.91 (Llama / Qwen) down to register α35 +1.33 / +1.07 centi-nats, i.e. about 1–2% perplexity.
- The ranking is the reverse of the benchmarks: wiki beats register on loss (register − wiki at α35: -0.63 / -0.59), mainly on arXiv and Wikipedia.
- α35 models are worse than base on FineWeb (-0.44 to -0.29).
Sources: ppl_deltas.csv, ppl_contrasts.csv; write-up PPL_FINDINGS.md.
3 · Where the loss gains come from
All 10 models scored on identical tokens (7 domains × 1,024 sequences × 2,048 tokens). Median over the 7 domains of the mean Δ per token group, centi-nats (L = Llama, Q = Qwen):
| token group | L register α15 | L register α35 | L wiki α15 | L wiki α35 | Q register α15 | Q register α35 | Q wiki α15 | Q wiki α35 |
|---|---|---|---|---|---|---|---|---|
| easiest 20% of tokens | -0.02 | +0.06 | -0.03 | -0.04 | +0.04 | +0.03 | -0.10 | -0.12 |
| hardest 20% of tokens | +2.98 | +4.02 | +3.78 | +4.95 | +3.05 | +4.38 | +5.67 | +3.81 |
| tokens unseen in base training corpus | +9.44 | +8.96 | +10.45 | +11.55 | +5.48 | +2.15 | +7.88 | +7.87 |
| 100 most frequent tokens | +0.84 | +0.84 | +0.81 | +0.26 | +0.38 | +0.37 | +0.17 | +0.07 |
| function words | +0.97 | +0.04 | +0.91 | +0.17 | +0.96 | -0.11 | +0.74 | -0.21 |
| whitespace / layout | +7.30 | +10.81 | +5.19 | +12.28 | +5.48 | +9.45 | +6.40 | +10.37 |
- Gains sit on hard and rare tokens and on whitespace; the easiest tokens are unchanged.
- α35 loses the gain on function words (and on words in FineWeb and FreeLaw, below), while its whitespace gains grow.
- On the hardest 20% of tokens, loss falls by 2.0–5.0 centi-nats while predictive entropy rises by at most 1.2 and in 6 of 8 models falls (by up to 2.5; medians over domains). The models rank the right token higher; they don't hedge more.
α35 predicts words worse than base on FineWeb and FreeLaw
| domain | words, α15 models | words, α35 models | whitespace, α35 models |
|---|---|---|---|
| FineWeb | +0.35 to +0.58 | -0.39 to -0.21 | -1.9 to -0.3 |
| FreeLaw | +0.21 to +0.84 | -1.10 to -0.50 | +12.3 to +16.6 |
| Gutenberg | +0.80 to +1.98 | -0.82 to +0.23 | +2.6 to +12.4 |
| arXiv | +3.25 to +4.42 | +1.82 to +5.28 | +9.4 to +17.2 |
Δ on non-whitespace tokens, derived from the total Δ and the whitespace share/Δ ((total − s·Δws)/(1 − s)); ranges over register and wiki in both architectures, centi-nats.
The steered training text is less predictable
NLL change of the synthetic training text vs the base corpus on common tokens, under two outside scorers (negative = less predictable): Qwen2.5-1.5B register α15 -0.11, register α35 -0.28, wiki α15 -0.11, wiki α35 -0.25; Pythia-1.4B register α15 -0.07, register α35 -0.21, wiki α15 -0.07, wiki α35 -0.19 nats. α35 corpora are 2–3× further from base than α15; register and wiki corpora are nearly identical. The less predictable corpora give the smaller loss gains (α35 < α15).
Sources: attribution (llama/qwen), token_overall.csv, token_special.csv, token_calib_quintiles.csv, corpus predictability (qwen/pythia); write-up FINDINGS_all_arms.md.
4 · Benchmark gains are on the answer, not on reading the input
All 10 models were re-scored on the exact lighteval prompts (3-shot) plus the gold answer, for 1,500 items each of WikiTableQ, SQuADv2 and DROP, all 490 TREB-QA, and 2,000 each of TriviaQA and LAMBADA. Our answer log-probabilities reproduce lighteval's per item (correlation ≥ 0.9999).
- The gold answer gains 3.8–11.2 centi-nats on average; the question and passage/table text of the same items gains only 0.6–2.4.
- Across items, how much better a model reads the context does not predict how much better it predicts the answer (|Spearman ρ| ≤ 0.10 in every model × task).
Register vs wiki at α35: better answers, worse reading
| task | Llama 1B: answer / context | Qwen2 1.7B: answer / context |
|---|---|---|
| WikiTableQ | +8.3† / -1.4† | +3.0† / -0.6† |
| DROP | +4.4† / -2.3† | +5.4† / -0.9† |
| SQuADv2 | +6.1† / -2.3† | +1.7 / -1.9† |
| TriviaQA | +4.1† / -2.2† | +3.9† / -2.8† |
| TREB-QA | +7.7† / -0.6† | -1.5 / -0.1 |
| LAMBADA | -8.6† / +0.1 | -4.7† / -0.3 |
Loss difference register α35 − wiki α35 (positive = register lower loss), centi-nats; † = BH q < 0.05.
Sources: bench_answer.csv, bench_segments.csv, bench_contrasts.csv, bench_link.csv, bench_sanity.csv; write-up FINDINGS_benchmark_and_format.md.
5 · What steering changes in the training answers
All synthetic corpora rewrite the same FineWeb documents as FAQs (SmolLM2-1.7B-Instruct, FAQ prompt). 15,000 source documents rewritten by every corpus, compared paired; document-format rows from 20,000 documents per corpus.
| base | register α15 | register α35 | wiki α15 | wiki α35 | |
|---|---|---|---|---|---|
| words per document | 320 | 337 | 416 | 314 | 390 |
| Q/A pairs per document | 5.8 | 6.1 | 6.9 | 6.0 | 6.7 |
| words per answer | 43.6 | 44.7 | 54.0 | 42.5 | 50.7 |
| answer opens with a short sentence (≤ 8 words) | 13.7% | 18.3% | 24.8% | 17.9% | 23.4% |
| answer first restates the question | 44.8% | 44.5% | 39.0% | 43.3% | 38.7% |
compact Q↵A: layout (benchmark style) | 43.6% | 40.4% | 33.1% | 40.5% | 38.4% |
| answer numbers not supported by the source | 19.6% | 27.1% | 41.9% | 27.7% | 43.7% |
| answer numbers that are list numbering | 10.9% | 11.5% | 10.4% | 10.1% | 7.5% |
- Answers get longer, but more of them open with a short direct reply (13.7% → 23.4–24.8% at α35) and fewer restate the question.
- Fewer documents use the compact benchmark layout after steering: steering does not make the data look more like the benchmarks.
- More answer numbers are added by the rewrite (not in the source document, after excluding list numbering, number words, reformatting and numbers from the question): 20% in base → 42–44% at α35. From reading examples: mostly invented details, a few true background facts.
- Register and wiki corpora are almost identical; their differences are 1–3 points:
| paired difference (points) | register − wiki, α15 | register − wiki, α35 |
|---|---|---|
| answer starts with yes/no | -1.4† | -3.1† |
| answer starts with a content word | +0.9† | +2.2† |
| answer opens with a short sentence | +0.5 | +1.8† |
| answer starts with words/number from the source | +1.2† | +1.7† |
† = 95% bootstrap CI over documents excludes 0. Benchmark gold answers are 1–2 words and almost never start with yes/no, so register's direction is closer to them, but the gap is small.
Sources: answers_content.csv, answers_contrasts.csv, answers_numbers.csv, corpus_format.csv; write-up FINDINGS_answer_content.md.
6 · Steered models prefer to answer with a short span
On 2,500 benchmark items (500 per QA task), the probability that the first token after Answer: starts a span (a number, name or word) rather than a sentence ("The", "It", "Yes", …), and the models' own greedy answers. 0-shot = question with its passage/table only; 3-shot = the benchmark prompt.
- Every steered model prefers spans more than its base, in both settings (31 of 32 differences significant, counting span probability and greedy answers). Without demos, base models answer with a sentence 89–91% of the time; α35 models 74–79%.
- Register > wiki in 4 of 4 comparisons at 0-shot, but only 1 of 4 at 3-shot.
- The preference roughly tracks the benchmark gains across the 8 models (Spearman ρ: P(span) 0.81 at 0-shot, 0.67 at 3-shot; greedy span answers 0.57 / 0.43). Llama wiki α35 is the clearest exception at 0-shot (second-largest span shift among Llama models, smallest Llama gain).
- Greedy answers do not contain the gold answer more often (15–19% for every model): what changes is the form of the answer and the probability of the gold span.
| L register α15 | L register α35 | L wiki α15 | L wiki α35 | Q register α15 | Q register α35 | Q wiki α15 | Q wiki α35 | |
|---|---|---|---|---|---|---|---|---|
| benchmark gain, 5 QA tasks (points) | +0.94 | +1.58 | +1.15 | +0.61 | +1.30 | +1.72 | +0.31 | +1.44 |
| Δ P(span), 0-shot (points) | +2.8 | +8.8 | +1.7 | +5.3 | +3.9 | +7.6 | +0.5 | +6.9 |
Sources: preference.csv, preference_vs_base.csv, bench_answer.csv; write-up FINDINGS_answer_content.md.
7 · What is shown and what is not
| statement | status | evidence |
|---|---|---|
| Steered models improve held-out loss; wiki more than register; α35 less than α15 | shown | §2 |
| Loss gains sit on hard/rare tokens and whitespace; α35 loses function words and prose words | shown | §3 |
| Benchmark gains are on the gold answer, not on reading the context | shown | §4 |
| At α35, training answers open more directly and contain more added numbers | shown | §5 |
| Register and wiki training data are nearly identical | shown | §3, §5 |
| Steered models prefer span answers (0-shot and 3-shot) | shown | §6 |
| Register models prefer spans more than wiki models | only 0-shot | §6 |
| The span preference follows the benchmark gains | partly (ρ 0.43–0.81, exceptions) | §6 |
| Register really beats wiki on benchmarks | not established (within seed noise) | §1 |
| The changed training answers cause the span preference, or the preference causes the gains | not shown (correlational, one seed) | — |
Caveats
- One training seed per model. CIs on this page cover evaluation/item noise only.
- α35 mixes also contain different human text (other CommonCrawl dumps, because longer synthetic documents shift which FineWeb shards fill the human half); this is collinear with α. Its measured effect on predictability is ~10× smaller than the synthetic difference, but it is not ruled out.
- Source-document faithfulness concerns a document the model never sees during training.
Next
A temperature-matched baseline (unsteered generator at T ≈ 1.08, matched to register α35's predictability) is being generated; it tests whether the effects come from the steering direction or just from less predictable text. Extra seeds would settle register vs wiki. See RUN_PLAN.md.