Steered paraphrase pretraining: what changes, and where the gains come from
Small LMs pretrained on FAQ paraphrases generated with activation steering (register or wiki direction). Results summary; every number below is computed from the CSVs in data/.
Summary
- Benchmarks: macro changes from +0.01 to +1.03 points (Qwen2 1.7B register α35; Qwen2 2.9B register α35 +0.71). Register leads wiki in all five comparisons. Gains on DROP, tables, XCSQA, GSM8K (Qwen); costs on LAMBADA and SQuADv2.
- Probability vs accuracy: at 1–1.7B the gain is mostly higher probability on answer-like text; at 2.9B it is more correct choices. Knowledge is unchanged.
- Language modelling: every steered model lowers held-out loss by ~1–2% perplexity, wiki most and α35 least: the reverse of the benchmark ranking.
- Mechanism (loss): gains on hard/rare tokens and layout; α35 gives up gains on function words and prose.
- Mechanism (benchmarks): gains are on the gold answer, not on reading the context. Steered models prefer short span answers, most at α35, and this roughly tracks the benchmark gains.
- Data: steering makes FAQ answers open more directly and contain more numbers that are not in the source document; register and wiki data are nearly identical.
- Vectors: more human sources give the same direction (cos 0.91–0.92 with register) with more cost. New contrasts are new directions; random directions do nothing to the text; register + a faithfulness correction removes the unsupported numbers while keeping the NLL shift.
- Generation: 3 vLLM engines per GPU, ~1.2× faster with the same output distribution.
Setup
- Models: Llama 1B and Qwen2 1.7B (base + four steered models each) and Qwen2 2.9B (base, register α35, wiki α35), pretrained from scratch, one seed each.
- Data: one epoch over a 20B-token mix, 50:50 synthetic and human (FineWeb). The synthetic half is FineWeb rewritten as FAQs by SmolLM2-1.7B-Instruct. Base: no steering. Register and wiki: a steering vector added at layer 14 with strength α = 15 or 35.
- Units: benchmark Δ in points (prob_norm_token × 100); loss Δ in centi-nats (nats × 100), positive = better than base. CIs are 95% paired bootstraps.
1 · Benchmarks: small gains; one clearly above seed noise
Paper-style evaluation (lighteval, 3-shot, cloze, prob_norm_token), every question of every task, paired bootstrap against each architecture's base. Shaded band: the finephrase paper's seed-to-seed SD of this score (0.16–0.49 points). One seed per model.
- Largest gain: Qwen2 1.7B register α35 +1.03 [+0.95, +1.11]. Qwen2 2.9B register α35 +0.71, wiki α35 +0.26. Base macro: Llama 16.44, Qwen 1.7B 17.53, Qwen 2.9B 18.66.
- The 2.9B models saw the same 20B tokens (≈7.5 tokens/parameter vs ≈12 at 1.7B), so they are the least trained.
Direct comparisons (macro Δ, points, 95% CI)
| contrast | Llama 1B | Qwen2 1.7B | Qwen2 2.9B |
|---|---|---|---|
| register − wiki, α15 | +0.12 [+0.06, +0.19] | +0.29 [+0.22, +0.36] | – |
| register − wiki, α35 | +0.38 [+0.31, +0.44] | +0.38 [+0.31, +0.46] | +0.45 [+0.38, +0.52] |
| α35 − α15, register | -0.06 [-0.13, +0.01] | +0.37 [+0.30, +0.45] | – |
| α35 − α15, wiki | -0.32 [-0.38, -0.26] | +0.28 [+0.21, +0.35] | – |
Register beats wiki in all five comparisons (widest at 2.9B: +0.45); single-seed margins below ~0.23–0.69 are within the noise expected between two models.
Per task
| WikiTableQ | TREB-QA | DROP | SQuADv2 | TriviaQA | XCSQA | ARC-Easy | ARC-Challenge | OpenBookQA | HellaSwag | PIQA | WinoGrande | MMLU-Redux | GSM8K | LAMBADA | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama 1B · register α15 | +0.50† | +0.09 | +2.13† | +1.50† | +0.17† | +0.88† | +0.56† | +0.40† | +0.18† | +0.15† | +0.23† | +0.39† | +0.28† | -0.55† | +0.24 |
| Llama 1B · register α35 | +2.52† | +0.88† | +2.81† | +0.54† | +0.69† | +1.21† | +0.84† | +0.33† | +0.28† | +0.07† | +0.08† | +0.28† | +0.33† | -2.35† | -0.86† |
| Llama 1B · wiki α15 | +0.39† | +0.94† | +1.51† | +1.36† | +0.53† | +0.70† | +0.54† | +0.31† | +0.12† | +0.04† | +0.25† | +0.57† | +0.06 | -1.11† | -0.26 |
| Llama 1B · wiki α35 | +0.84† | -0.16 | +2.26† | -0.94† | +0.79† | +1.28† | +1.04† | +0.35† | +0.20† | +0.09† | +0.34† | +0.42† | +0.35† | -2.51† | +0.25 |
| Qwen2 1.7B · register α15 | +2.35† | +0.96† | +2.18† | +0.51† | +0.27† | +0.10 | +0.33† | +0.23† | +0.20† | +0.18† | +0.09† | -0.04 | +0.14† | +0.47† | +0.05 |
| Qwen2 1.7B · register α35 | +4.32† | +1.65† | +2.75† | -0.10 | +0.09 | +0.60† | +0.42† | +0.37† | +0.53† | +0.15† | +0.00 | +0.34† | +0.28† | +0.87† | -0.86† |
| Qwen2 1.7B · wiki α15 | +1.90† | +0.91† | -0.71† | -0.96† | +0.15† | +0.56† | +0.48† | +0.30† | +0.38† | +0.23† | +0.12† | +0.04 | +0.13 | +0.80† | +0.53† |
| Qwen2 1.7B · wiki α35 | +3.43† | +1.60† | +2.77† | -1.57† | +0.22† | +0.34† | +0.41† | +0.38† | +0.42† | +0.21† | +0.19† | +0.28† | -0.00 | -0.03 | -0.08 |
| Qwen2 2.9B · register α35 | +1.70† | +0.13 | +4.99† | -1.31† | +0.43† | +1.41† | -0.07 | +0.11 | +0.06 | -0.05† | -0.00 | -0.43† | +0.10 | +1.25† | -1.06† |
| Qwen2 2.9B · wiki α35 | -0.17† | -0.41 | +2.23† | -1.55† | +0.52† | +1.06† | +0.45† | +0.06 | +0.14 | +0.06† | +0.14† | -0.40† | +0.08 | +0.96† | -0.01 |
Δ prob_norm_token × 100 vs base, full evaluation sets. † = significant after BH correction (q < 0.05). Blue = better than base, red = worse. Hover a cell for its value.
- 105 of 150 model × task changes are significantly positive, 17 significantly negative.
- Gains: DROP (significant in 9 of 10 models, up to +4.99), WikiTableQ (9 of 10; -0.17 to +4.32), XCSQA (positive in all 10, up to +1.41), GSM8K at both Qwen sizes, TriviaQA at 2.9B.
- Costs: LAMBADA for register α35 in all three families (-0.86, -0.86, -1.06), SQuADv2 at 2.9B, GSM8K on Llama (down to -2.51). Knowledge tasks (MMLU-Redux, ARC-Challenge, OpenBookQA) barely move.
Probability vs accuracy: what the models actually learn
Only the 8 multiple-choice tasks have an accuracy (acc_norm_token). The two metrics answer different questions: probability = how much mass the correct answer gets; accuracy = whether it beats the wrong options. Probability up with accuracy flat means the wrong options rose too (a general boost to answer-like text). Accuracy up with probability flat means better choices without more confidence.
| task (base acc %: L / Q1.7 / Q2.9) | L reg α15 | L reg α35 | L wiki α15 | L wiki α35 | Q1.7 reg α15 | Q1.7 reg α35 | Q1.7 wiki α15 | Q1.7 wiki α35 | Q2.9 reg α35 | Q2.9 wiki α35 |
|---|---|---|---|---|---|---|---|---|---|---|
| XCSQA (39 / 45 / 47) | +2.9† / +0.88† | +5.9† / +1.21† | +4.0† / +0.70† | +4.5† / +1.28† | +0.1 / +0.10 | +2.8 / +0.60† | +2.7 / +0.56† | +1.4 / +0.34† | +4.4† / +1.41† | +0.8 / +1.06† |
| HellaSwag (43 / 46 / 50) | +1.0† / +0.15† | +0.9† / +0.07† | +0.6 / +0.04† | +0.7 / +0.09† | +0.9† / +0.18† | +1.5† / +0.15† | +0.8 / +0.23† | +1.5† / +0.21† | +1.0† / -0.05† | +1.5† / +0.06† |
| ARC-Easy (52 / 57 / 58) | +1.0 / +0.56† | +1.6 / +0.84† | +2.4† / +0.54† | +2.5† / +1.04† | +1.3 / +0.33† | +0.8 / +0.42† | +1.1 / +0.48† | +1.3 / +0.41† | +3.1† / -0.07 | +3.1† / +0.45† |
| WinoGrande (53 / 52 / 54) | -0.6 / +0.39† | +0.5 / +0.28† | -0.6 / +0.57† | -0.9 / +0.42† | +1.0 / -0.04 | +0.9 / +0.34† | -0.2 / +0.04 | +1.3 / +0.28† | -1.0 / -0.43† | -1.0 / -0.40† |
| PIQA (71 / 72 / 73) | +0.8 / +0.23† | +0.9 / +0.08† | +1.9† / +0.25† | +1.0 / +0.34† | +1.1 / +0.09† | +0.3 / +0.00 | +0.4 / +0.12† | +0.7 / +0.19† | +0.8 / -0.00 | +0.7 / +0.14† |
| ARC-Challenge (26 / 27 / 30) | +0.2 / +0.40† | +1.1 / +0.33† | -1.0 / +0.31† | +0.3 / +0.35† | -0.6 / +0.23† | +1.7 / +0.37† | +0.5 / +0.30† | +1.4 / +0.38† | -0.4 / +0.11 | +0.7 / +0.06 |
| OpenBookQA (26 / 28 / 28) | +0.2 / +0.18† | +0.8 / +0.28† | +0.2 / +0.12† | -0.2 / +0.20† | -1.0 / +0.20† | +2.6 / +0.53† | -1.0 / +0.38† | +1.0 / +0.42† | +0.0 / +0.06 | -0.4 / +0.14 |
| MMLU-Redux (30 / 31 / 32) | +0.3 / +0.28† | +0.4 / +0.33† | +0.2 / +0.06 | +0.8 / +0.35† | +0.1 / +0.14† | -0.1 / +0.28† | -0.3 / +0.13 | +0.3 / -0.00 | -1.0 / +0.10 | +0.0 / +0.08 |
| significant gains / losses, probability | 8 / 0 | 8 / 0 | 7 / 0 | 8 / 0 | 6 / 0 | 7 / 0 | 6 / 0 | 7 / 0 | 1 / 2 | 4 / 1 |
| significant gains / losses, accuracy | 2 / 0 | 2 / 0 | 3 / 0 | 2 / 0 | 1 / 0 | 1 / 0 | 0 / 0 | 1 / 0 | 3 / 0 | 2 / 0 |
Accuracy Δ / probability Δ, points; † = BH q < 0.05. Accuracy needs ~2–3 points to be significant on these set sizes (500–2,400 questions; HellaSwag 10,042).
- Llama 1B and Qwen 1.7B: mostly a format effect. Probability rises on 6–8 of 8 tasks, accuracy on only 0–3 (XCSQA, HellaSwag).
- Qwen 2.9B: better choices. Probability is flat, but register α35 accuracy rises on ARC-Easy (+3.1), XCSQA (+4.4) and HellaSwag (+1.0, while its probability falls).
- The prose cost is lost confidence, not worse judgement (HellaSwag at 2.9B); WinoGrande at 2.9B is the only task where both point down.
- In short: better at turning a question into a short, specific answer; worse at plain-prose likelihood and exact span copying; knowledge unchanged. The size pattern rests on one seed per size.
Sources: aggregates, task deltas, contrasts (and the qwen, qwen29 files); write-up LIGHTEVAL_NALL.md (findings 6–7).
2 · Held-out loss: every model improves, wiki most, α35 least
Per-domain held-out loss, recomputed on the full evaluation splits (the final_eval.json written at the end of training held the FineWeb loss under every domain name). Paired bootstrap over sequences.
| FineWeb | Wikipedia | arXiv | HackerNews | StackExchange | FreeLaw | Gutenberg | Mean | |
|---|---|---|---|---|---|---|---|---|
| Llama 1B · register α15 | +0.62† | +1.38† | +3.70† | +1.73† | +2.90† | +1.08† | +1.63† | +1.86† |
| Llama 1B · register α35 | -0.29† | +1.27† | +2.75† | +2.52† | +2.54† | +0.20† | +0.34† | +1.33† |
| Llama 1B · wiki α15 | +0.32† | +1.78† | +4.80† | +1.74† | +2.99† | +1.13† | +1.49† | +2.04† |
| Llama 1B · wiki α35 | -0.36† | +2.90† | +5.60† | +1.84† | +2.95† | +0.35† | +0.48† | +1.96† |
| Qwen2 1.7B · register α15 | +0.51† | +1.26† | +3.22† | +1.62† | +2.66† | +1.10† | +1.22† | +1.66† |
| Qwen2 1.7B · register α35 | -0.44† | +1.03† | +2.17† | +2.02† | +2.18† | +0.18† | +0.33† | +1.07† |
| Qwen2 1.7B · wiki α15 | +0.40† | +1.69† | +4.20† | +1.11† | +3.10† | +1.33† | +1.54† | +1.91† |
| Qwen2 1.7B · wiki α35 | -0.40† | +2.50† | +4.80† | +1.11† | +2.83† | +0.52† | +0.25† | +1.66† |
Δ loss vs base in centi-nats (nats × 100) on the full held-out splits; positive = lower loss than base. † = BH q < 0.05.
- Mean gain: wiki α15 +2.04 / +1.91 (Llama / Qwen) down to register α35 +1.33 / +1.07 centi-nats, i.e. about 1–2% perplexity.
- The ranking is the reverse of the benchmarks: wiki beats register on loss (register − wiki at α35: -0.63 / -0.59), mainly on arXiv and Wikipedia.
- α35 models are worse than base on FineWeb (-0.44 to -0.29).
- Qwen2 2.9B (from its end-of-training
final_eval.json, not in the table): mean gain register α35 +0.96, wiki α35 +1.43, FineWeb −0.77 for both. The benefit shrinks gradually with size (Llama → Qwen 1.7B → 2.9B).
Sources: ppl_deltas.csv, ppl_contrasts.csv; write-up PPL_FINDINGS.md.
3 · Where the loss gains come from
All 10 models scored on identical tokens (7 domains × 1,024 sequences × 2,048 tokens). Median over the 7 domains of the mean Δ per token group, centi-nats (L = Llama, Q = Qwen):
| token group | L register α15 | L register α35 | L wiki α15 | L wiki α35 | Q register α15 | Q register α35 | Q wiki α15 | Q wiki α35 |
|---|---|---|---|---|---|---|---|---|
| easiest 20% of tokens | -0.02 | +0.06 | -0.03 | -0.04 | +0.04 | +0.03 | -0.10 | -0.12 |
| hardest 20% of tokens | +2.98 | +4.02 | +3.78 | +4.95 | +3.05 | +4.38 | +5.67 | +3.81 |
| tokens unseen in base training corpus | +9.44 | +8.96 | +10.45 | +11.55 | +5.48 | +2.15 | +7.88 | +7.87 |
| 100 most frequent tokens | +0.84 | +0.84 | +0.81 | +0.26 | +0.38 | +0.37 | +0.17 | +0.07 |
| function words | +0.97 | +0.04 | +0.91 | +0.17 | +0.96 | -0.11 | +0.74 | -0.21 |
| whitespace / layout | +7.30 | +10.81 | +5.19 | +12.28 | +5.48 | +9.45 | +6.40 | +10.37 |
- Gains sit on hard and rare tokens and on whitespace; the easiest tokens are unchanged.
- α35 loses the gain on function words (and on words in FineWeb and FreeLaw, below), while its whitespace gains grow.
- On the hardest 20% of tokens, loss falls by 2.0–5.0 centi-nats while predictive entropy rises by at most 1.2 and in 6 of 8 models falls (by up to 2.5; medians over domains). The models rank the right token higher; they don't hedge more.
α35 predicts words worse than base on FineWeb and FreeLaw
| domain | words, α15 models | words, α35 models | whitespace, α35 models |
|---|---|---|---|
| FineWeb | +0.35 to +0.58 | -0.39 to -0.21 | -1.9 to -0.3 |
| FreeLaw | +0.21 to +0.84 | -1.10 to -0.50 | +12.3 to +16.6 |
| Gutenberg | +0.80 to +1.98 | -0.82 to +0.23 | +2.6 to +12.4 |
| arXiv | +3.25 to +4.42 | +1.82 to +5.28 | +9.4 to +17.2 |
Δ on non-whitespace tokens, derived from the total Δ and the whitespace share/Δ ((total − s·Δws)/(1 − s)); ranges over register and wiki in both architectures, centi-nats.
The steered training text is less predictable
NLL change of the synthetic training text vs the base corpus on common tokens, under two outside scorers (negative = less predictable): Qwen2.5-1.5B register α15 -0.11, register α35 -0.28, wiki α15 -0.11, wiki α35 -0.25; Pythia-1.4B register α15 -0.07, register α35 -0.21, wiki α15 -0.07, wiki α35 -0.19 nats. α35 corpora are 2–3× further from base than α15; register and wiki corpora are nearly identical. The less predictable corpora give the smaller loss gains (α35 < α15).
Sources: attribution (llama/qwen), token_overall.csv, token_special.csv, token_calib_quintiles.csv, corpus predictability (qwen/pythia); write-up FINDINGS_all_arms.md.
4 · Benchmark gains are on the answer, not on reading the input
All 10 models were re-scored on the exact lighteval prompts (3-shot) plus the gold answer, for 1,500 items each of WikiTableQ, SQuADv2 and DROP, all 490 TREB-QA, and 2,000 each of TriviaQA and LAMBADA. Our answer log-probabilities reproduce lighteval's per item (correlation ≥ 0.9999).
- The gold answer gains 3.8–11.2 centi-nats on average; the question and passage/table text of the same items gains only 0.6–2.4.
- Across items, how much better a model reads the context does not predict how much better it predicts the answer (|Spearman ρ| ≤ 0.10 in every model × task).
Register vs wiki at α35: better answers, worse reading
| task | Llama 1B: answer / context | Qwen2 1.7B: answer / context |
|---|---|---|
| WikiTableQ | +8.3† / -1.4† | +3.0† / -0.6† |
| DROP | +4.4† / -2.3† | +5.4† / -0.9† |
| SQuADv2 | +6.1† / -2.3† | +1.7 / -1.9† |
| TriviaQA | +4.1† / -2.2† | +3.9† / -2.8† |
| TREB-QA | +7.7† / -0.6† | -1.5 / -0.1 |
| LAMBADA | -8.6† / +0.1 | -4.7† / -0.3 |
Loss difference register α35 − wiki α35 (positive = register lower loss), centi-nats; † = BH q < 0.05.
Sources: bench_answer.csv, bench_segments.csv, bench_contrasts.csv, bench_link.csv, bench_sanity.csv; write-up FINDINGS_benchmark_and_format.md.
5 · What steering changes in the training answers
All synthetic corpora rewrite the same FineWeb documents as FAQs (SmolLM2-1.7B-Instruct, FAQ prompt). 15,000 source documents rewritten by every corpus, compared paired; document-format rows from 20,000 documents per corpus.
| base | register α15 | register α35 | wiki α15 | wiki α35 | |
|---|---|---|---|---|---|
| words per document | 320 | 337 | 416 | 314 | 390 |
| Q/A pairs per document | 5.8 | 6.1 | 6.9 | 6.0 | 6.7 |
| words per answer | 43.6 | 44.7 | 54.0 | 42.5 | 50.7 |
| answer opens with a short sentence (≤ 8 words) | 13.7% | 18.3% | 24.8% | 17.9% | 23.4% |
| answer first restates the question | 44.8% | 44.5% | 39.0% | 43.3% | 38.7% |
compact Q↵A: layout (benchmark style) | 43.6% | 40.4% | 33.1% | 40.5% | 38.4% |
| answer numbers not supported by the source | 19.6% | 27.1% | 41.9% | 27.7% | 43.7% |
| answer numbers that are list numbering | 10.9% | 11.5% | 10.4% | 10.1% | 7.5% |
- Answers get longer, but more of them open with a short direct reply (13.7% → 23.4–24.8% at α35) and fewer restate the question.
- Fewer documents use the compact benchmark layout after steering: steering does not make the data look more like the benchmarks.
- More answer numbers are added by the rewrite (not in the source document, after excluding list numbering, number words, reformatting and numbers from the question): 20% in base → 42–44% at α35. From reading examples: mostly invented details, a few true background facts.
- Register and wiki corpora are almost identical; their differences are 1–3 points:
| paired difference (points) | register − wiki, α15 | register − wiki, α35 |
|---|---|---|
| answer starts with yes/no | -1.4† | -3.1† |
| answer starts with a content word | +0.9† | +2.2† |
| answer opens with a short sentence | +0.5 | +1.8† |
| answer starts with words/number from the source | +1.2† | +1.7† |
† = 95% bootstrap CI over documents excludes 0. Benchmark gold answers are 1–2 words and almost never start with yes/no, so register's direction is closer to them, but the gap is small.
Sources: answers_content.csv, answers_contrasts.csv, answers_numbers.csv, corpus_format.csv; write-up FINDINGS_answer_content.md.
6 · Steered models prefer to answer with a short span
On 2,500 benchmark items (500 per QA task), the probability that the first token after Answer: starts a span (a number, name or word) rather than a sentence ("The", "It", "Yes", …), and the models' own greedy answers. 0-shot = question with its passage/table only; 3-shot = the benchmark prompt.
- Every steered model prefers spans more than its base, in both settings (31 of 32 differences significant, counting span probability and greedy answers). Without demos, base models answer with a sentence 89–91% of the time; α35 models 74–79%.
- Register > wiki in 4 of 4 comparisons at 0-shot, but only 1 of 4 at 3-shot.
- The preference roughly tracks the benchmark gains across the 8 models (Spearman ρ: P(span) 0.81 at 0-shot, 0.67 at 3-shot; greedy span answers 0.57 / 0.43). Llama wiki α35 is the clearest exception at 0-shot (second-largest span shift among Llama models, smallest Llama gain).
- Greedy answers do not contain the gold answer more often (15–19% for every model): what changes is the form of the answer and the probability of the gold span.
| L register α15 | L register α35 | L wiki α15 | L wiki α35 | Q register α15 | Q register α35 | Q wiki α15 | Q wiki α35 | |
|---|---|---|---|---|---|---|---|---|
| benchmark gain, 5 QA tasks (points) | +0.94 | +1.58 | +1.15 | +0.61 | +1.30 | +1.72 | +0.31 | +1.44 |
| Δ P(span), 0-shot (points) | +2.8 | +8.8 | +1.7 | +5.3 | +3.9 | +7.6 | +0.5 | +6.9 |
Sources: preference.csv, preference_vs_base.csv, bench_answer.csv; write-up FINDINGS_answer_content.md.
7 · Steering vectors: more sources, new contrasts
Same contrast, more sources. 16 human sources written before LLM text (Pile: wiki, gutenberg, freelaw, arxiv, pubmed, hackernews, enron, stackexchange; CC-News; Reddit; 6 CORE registers), 4,000 windows of 200–800 tokens each. Each window was paraphrased by SmolLM2-1.7B-Instruct (neutral prompt, best of 10 samples; length ratio 0.85–1.18, ≤ 50% 5-gram overlap, no prompt echo), ≤ 2,000 pairs per source. Vector = whitened mean difference (human − paraphrase) of mean-pooled layer-14 activations, unit norm, the same construction as register. Split-half reliability 0.986–0.997; per-source cosine with register 0.45 (gutenberg) to 0.88 (wiki). register and wiki are the existing vectors from the earlier pipeline (used for all trained models); the composites average the new per-source vectors (equal weight per source):
| vector | human sources (registers) |
|---|---|
register (existing) | 5 registers, 500 samples each: FineWeb, and Pile (monology) PubMed, HackerNews, StackExchange, Wikipedia |
wiki (existing) | Pile (monology) Wikipedia |
all_mean_unit | all 16: Wikipedia, Gutenberg books, FreeLaw court opinions, arXiv papers, PubMed abstracts, HackerNews comments, Enron e-mails, StackExchange Q&A, CC-News articles, Reddit posts, CORE web registers (narrative, opinion, informational, interactive discussion, how-to, informational persuasion) |
no_outliers | the same without Gutenberg, FreeLaw and StackExchange (13) |
expository | Wikipedia, CC-News, arXiv, PubMed, CORE informational |
New contrasts, each aimed at one candidate ingredient (same construction; output-based contrasts are encoded as the steered model sees them: FAQ prompt + source + rewrite, pooling only the rewrite/answer):
| contrast | toward (+) | away from (−) | activations | reliability | held-out sep. | cos register |
|---|---|---|---|---|---|---|
| edu_quality | FineWeb windows, FineWeb-Edu score ≥ 3 | score < 1, length-matched | plain text | 0.97 | 1.00 | -0.01 |
| formulaicity | least predictable of 8 unsteered rewrites (Qwen + Pythia NLL) | most predictable rewrite, same document | generation context | 0.97 | 0.94 | -0.13 |
| temperature | T 1.08 corpus rewrite | base rewrite, same document | generation context | 0.85 | 0.71 | -0.13 |
| answer_style | short direct answer that opens with a source span | answer restating the question, same rewrite | first sentence of the answer | 0.96 | 0.92 | -0.00 |
| faithfulness | answer whose numbers are all in the source | answer with an unsupported number, same rewrite | generation context | 0.94 | 0.80 | -0.04 |
| realized_reg35 | register α35 corpus rewrite | base rewrite, same document | diagnostic: what steering actually changed in the outputs | 0.98 | 0.96 | +0.32 |
Held-out separation = pair accuracy (paired) or AUROC (edu) on a held-out half. reg_plus_faith = register + 0.5 × faithfulness (orthogonalized), α chosen to keep register's push at 35. Random controls: Gaussian directions at the same norm.
Cosines (layer 14)
| vector | register | wiki | all_mean_unit | formulaicity | temperature | faithfulness |
|---|---|---|---|---|---|---|
| all_mean_unit | 0.91 | 0.67 | 1.00 | -0.17 | -0.21 | -0.08 |
| no_outliers | 0.91 | 0.68 | 1.00 | -0.17 | -0.20 | -0.07 |
| expository | 0.92 | 0.74 | 0.95 | -0.17 | -0.20 | -0.08 |
| edu_quality | -0.01 | 0.10 | -0.02 | 0.01 | 0.00 | 0.01 |
| formulaicity | -0.13 | -0.07 | -0.17 | 1.00 | 0.53 | 0.09 |
| temperature | -0.13 | -0.08 | -0.21 | 0.53 | 1.00 | -0.43 |
| answer_style | -0.00 | 0.01 | 0.04 | -0.15 | -0.14 | 0.16 |
| faithfulness | -0.04 | -0.05 | -0.08 | 0.09 | -0.43 | 1.00 |
| realized_reg35 | 0.32 | 0.26 | 0.32 | -0.09 | 0.36 | -0.54 |
| reg_plus_faith | 0.89 | 0.71 | 0.79 | -0.08 | -0.31 | 0.41 |
| random_iso_s0 | 0.02 | -0.00 | 0.03 | 0.05 | 0.03 | -0.01 |
- The human-vs-paraphrase space is almost one-dimensional: every composite has 0.91–0.92 with register.
- All new contrasts are new directions (|cos| ≤ 0.13 with register). Formulaicity is not register (−0.13) but shares a direction with temperature (+0.53).
realized_reg35(register α35 rewrites − base rewrites of the same documents, i.e. what steering actually changed in the outputs) has only 0.32 with register itself and -0.54 with faithfulness.
Screen: what each vector does to the training text
The first 12,000 FineWeb documents of shard 0 rewritten with each vector (production FAQ setup, layer 14), compared document by document with the base corpus. Existing corpora and regenerated arms (*_regen) give the scale.
| set (vs base corpus) | Qwen NLL | Pythia NLL | tokens | capped % | FAQ docs % | short open % | restates Q % | source-span lead % | unsupported nums % | distinct bigrams | edu score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| corpus: reg35 | +0.23 | +0.16 | +134 | +7.6 | -7.8 | +12.6 | -5.2 | -14.8 | +20.6 | -0.008 | -0.05 |
| corpus: wiki35 | +0.18 | +0.12 | +105 | +6.5 | -8.6 | +10.8 | -6.2 | -15.6 | +22.9 | +0.003 | -0.05 |
| corpus: temp108 | +0.22 | +0.20 | +1ns | +0.2ns | -0.7ns | +0.5ns | -5.8 | -4.1 | +2.1 | +0.022 | -0.01 |
| base_regen | -0.02 | -0.01ns | +2ns | -0.1ns | -0.2ns | -0.3ns | +0.2ns | -0.4ns | -1.2ns | +0.000ns | +0.00ns |
| reg35_regen (α35) | +0.23 | +0.16 | +133 | +7.3 | -8.2 | +12.6 | -5.2 | -14.5 | +20.9 | -0.009 | -0.04 |
| all_mean_unit (α35) | +0.20 | +0.14 | +200 | +10.2 | -4.5 | +13.1 | -4.0 | -18.0 | +25.2 | -0.011 | -0.04 |
| no_outliers (α35) | +0.19 | +0.13 | +208 | +10.8 | -5.7 | +12.9 | -3.4 | -18.5 | +24.4 | -0.014 | -0.05 |
| expository (α35) | +0.16 | +0.10 | +214 | +10.1 | -3.9 | +12.2 | -2.7 | -17.6 | +24.7 | -0.016 | -0.04 |
| random_iso_s0 (α35) | -0.03 | -0.03 | +7 | -0.1ns | +2.1 | -1.5 | +4.0 | -6.1 | -0.9ns | -0.006 | +0.05 |
| random_whitened_s0 (α35) | -0.01ns | -0.01ns | +20 | +0.1ns | +2.7 | -1.9 | +3.2 | -5.7 | -0.8ns | -0.007 | +0.04 |
| reg_plus_faith (α39.1) | +0.20 | +0.13 | -44 | +1.4 | -12.4 | +12.1 | -3.8 | +3.2 | -0.6ns | -0.020 | -0.10 |
| formulaicity (α15.3) | +0.40 | +0.36 | -58 | -0.2 | -1.7 | +2.5 | -17.5 | -4.1 | -4.4 | +0.043 | -0.04 |
| edu_quality (α34.4) | +0.18 | +0.15 | -47 | -0.2ns | -6.2 | +1.3 | -4.3 | -12.3 | +5.0 | +0.031 | +0.40 |
| temperature (α16.1) | +0.59 | +0.51 | +34 | +1.4 | -1.2 | +5.2 | -19.7 | -17.7 | +9.6 | +0.053 | -0.01ns |
| answer_style (α16.2) | +0.08 | +0.06 | -17 | +0.0ns | -2.0 | +4.3 | -5.5 | +2.5 | -4.4 | +0.003 | -0.04 |
Change vs base corpus; NLL in nats/token under outside scorers (higher = less predictable); percentages in points. ns = 95% paired bootstrap CI includes 0. Base corpus values (contrasts/screen/SCREEN.md): Qwen NLL 1.96, 452 tokens, 74% FAQ docs, 20% unsupported answer numbers, edu 1.52.
How α was chosen: same-contrast composites and random controls use α35, like register (all vectors are unit norm). New contrasts use α = 35 × (spread of activations along the new direction ÷ spread along register), measured on SmolLM2 paraphrase activations at layer 14, so each pushes as far as register relative to natural variation. It's a rough yardstick: formulaicity at α15.3 shifts NLL almost twice as much as register. reg_plus_faith uses α39.1, which keeps its push along register at 35.
An α sweep (formulaicity α8/α11, edu α25/α45) is running to pick the final α by effect size.
- The screen reproduces the corpora:
reg35_regenmatches the register α35 corpus on every measure. - The direction matters: random directions at the same norm barely change the text (-0.03 NLL vs +0.23).
- More sources of the same contrast = register with more cost: all_mean_unit / no_outliers / expository give register's profile with longer outputs (+200 to +214 vs +134 tokens), more cap hits and more unsupported numbers.
- reg_plus_faith keeps register's NLL shift and short openings but brings unsupported numbers back to base (-0.6 vs +20.6 points) and removes the length inflation; price: -12.4 points of FAQ-shaped documents.
- formulaicity gives the largest NLL shift with fewer unsupported numbers than base and much less question-restating, but almost no short-opening effect: it separates "diversity" from register's answer style.
- edu_quality raises the outputs' edu score (+0.40; register lowers it). temperature overshoots and costs faithfulness; answer_style is weak at its α.
Sources: vector_sources.csv, vector_cosines.csv, contrast_reports.csv, screen.csv; write-ups vector_computation/results/COMPARISON.md, contrasts/README.md.
8 · Faster corpus generation
Generation (vLLM, SmolLM2-1.7B on GH200) now runs 3 engines per GPU (memory share 0.30 each, 768 concurrent sequences, suffix speculative decoding) instead of one large engine. A sweep of serving settings found this the fastest: 1.19× (steered) and 1.24× (unsteered) more output per GPU. A 120k-document test reproduced it: 7,750 vs 6,681 output tokens/s per GPU (1.16×), with the same output distribution (579 vs 579 tokens per document, 8.5% vs 8.6% hitting the length cap; every metric of the distribution check within 3–11% of the steering effect). Each shard takes ~2.5× longer because three share a GPU, so jobs are sized at 36 shards per 3 nodes (one round, ~10–13 h). About a sixth fewer GPU-hours per corpus.
Sources: optimized_inference/README.md, throughput_sweep/results/SUMMARY.md, throughput_sweep/VALIDITY.md.
9 · What is shown and what is not
| statement | status | evidence |
|---|---|---|
| Steered models improve held-out loss; wiki more than register; α35 less than α15 | shown | §2 |
| Loss gains sit on hard/rare tokens and whitespace; α35 loses function words and prose words | shown | §3 |
| Benchmark gains are on the gold answer, not on reading the context | shown | §4 |
| At α35, training answers open more directly and contain more added numbers | shown | §5 |
| Register and wiki training data are nearly identical | shown | §3, §5 |
| Steered models prefer span answers (0-shot and 3-shot) | shown | §6 |
| Register models prefer spans more than wiki models | only 0-shot | §6 |
| The span preference follows the benchmark gains | partly (ρ 0.43–0.81, exceptions) | §6 |
| Smaller models mostly gain answer probability; 2.9B gains accuracy (better choices) | suggestive (one seed per size) | §1 |
| The steering effect on the training text is specific to the direction (random ≈ base) | shown (corpus level) | §7 |
| A faithfulness correction removes the unsupported numbers and keeps the NLL shift | shown (corpus level) | §7 |
| New contrasts (formulaicity, edu, reg_plus_faith) change benchmarks | not tested yet | §7 |
| Register really beats wiki on benchmarks | not established (within seed noise) | §1 |
| The changed training answers cause the span preference, or the preference causes the gains | not shown (correlational, one seed) | — |
Caveats
- One training seed per model. CIs on this page cover evaluation/item noise only.
- α35 mixes also contain different human text (other CommonCrawl dumps, because longer synthetic documents shift which FineWeb shards fill the human half); this is collinear with α. Its measured effect on predictability is ~10× smaller than the synthetic difference, but it is not ruled out.
- Source-document faithfulness concerns a document the model never sees during training.
Next
- Seeds (Qwen base and register α35, one more at 2.9B): every comparison above is judged against a seed SD we haven't measured.
- Full arms from the screen:
reg_plus_faith(register without its faithfulness and length costs) andformulaicity(diversity alone);edu_qualityif budget allows. Not worth arms: the same-contrast composites, temperature vector, random (≈ base at corpus level). - Temperature-matched baseline (T 1.08 corpus, generated): does less predictable text alone give the gains?