# Held-out perplexity per domain: findings

The full tables are in [`REPORT.md`](REPORT.md), generated by `analyze_ppl.py`. Corrected numbers for each model are in `trained_models/<model>/<run>/final_eval_fixed.json`. The old `final_eval.json` files are left untouched.

## What was wrong

**Problem.** Each `final_eval.json` repeats the **FineWeb** loss under all 7 domain names.

**Cause.** In `code/pretrain_1b.py:300`, `dataloader_persistent_workers=True` is set. When `trainer.evaluate()` receives a dataset object rather than a name, the Trainer caches the dataloader under the key `"eval"`. The final loop (`:350`) therefore reused the FineWeb loader for every domain. The in-training evals passed names, so the wandb curves are fine.

**Fix for future runs.** Do one of these before the final loop:
- set `trainer.args.dataloader_persistent_workers = False`, or
- `del trainer._eval_dataloaders`, or
- call `trainer.evaluate(eval_dataset=name)` with the dict key.

**Recomputation.** `compute_domain_ppl.py` re-scores every packed 2,048-token sequence of each split. It uses labels = input_ids with nothing masked, and fp32 weights under bf16 autocast. SLURM job 210265 took 7–11 minutes per model.

**Sanity check.** The recomputed FineWeb loss reproduces the old value for all 10 models, up to a constant offset of +0.0014 nats. That offset is the same for every model; it comes from numerics (training-time torch.compile and the Trainer's autocast path), so the comparisons are unaffected.

## Findings

All Δ values are loss(base) − loss(model) in **centi-nats** (nats × 100), so positive means better than base. All are paired over identical sequences.

1. **Every intervention lowers held-out loss in both architectures.** The macro gain is +1.1 to +2.0 centi-nats, which is about 1–2% perplexity. Each domain has 1.6k–9k sequences, so every macro CI is narrow (±0.03–0.05). As with the benchmarks, this measures evaluation noise only; seed noise is still unmeasured.

   | Model | llama_1b macro Δ | qwen2_1.7b macro Δ |
   |---|---:|---:|
   | wiki_a15 | **+2.04** | **+1.91** |
   | wiki_a35 | +1.96 | +1.66 |
   | register_a15 | +1.86 | +1.66 |
   | register_a35 | +1.33 | +1.07 |

   The ranking is the same in both architectures: wiki_a15 ≥ wiki_a35 ≈ register_a15 > register_a35.

2. **For language modelling, wiki data beats register data.** The direct contrast register − wiki is negative at both α in both architectures: llama −0.17 / −0.63, qwen −0.25 / −0.59.
   - The wiki advantage comes from **Wikipedia (−0.4 to −1.6) and arXiv (−1.0 to −2.9)**: the encyclopedic and technical domains, which fits the wiki steering vectors.
   - Register wins only on **HackerNews** (+0.5 to +0.9 at α35) and slightly on FineWeb at α15.
   - **This is the reverse of the lighteval benchmark ranking**, where register beat wiki in all four comparisons. Lower perplexity and better benchmark scores are not explained by the same data property.

3. **α35 is worse than α15 for language modelling,** with a negative macro contrast in all 4 comparisons. Register shows the larger drop (−0.53 llama, −0.59 qwen); wiki is −0.07 and −0.25.
   - **Both α35 models are worse than base on FineWeb in both architectures** (−0.29 to −0.44 centi-nats, all †). They gain almost nothing on FreeLaw and Gutenberg (+0.2 to +0.5, against +1.1 to +1.6 at α15).
   - α35 helps only where the steering source matches the domain: HackerNews for register, Wikipedia and arXiv for wiki (wiki_a35 has the best Wikipedia and arXiv perplexity of all models).
   - **Caveat:** the α35 mixes also have a different human half. Their FineWeb shards come from different crawl dumps, with a total variation distance of 0.31–0.38 from base vs ≤ 0.07 for α15. FineWeb, the domain where α35 is worse than base, is also the domain most likely to react to that. See `loss_attribution_analysis/RUN_PLAN.md`.

4. **The `<eos>` fingerprint doesn't drive perplexity.** Excluding end-of-document targets changes every macro Δ by at most +0.1. (The FineWeb eval split contains no EOS tokens at all.)

5. **This is consistent with the preliminary loss attribution.**
   - For llama register_a15, the domain ordering matches `preliminary_findings.md`: arXiv > Stack > HackerNews ≈ Gutenberg > Wikipedia > FreeLaw > FineWeb.
   - Measured on the full splits, the gains are +3.7 / +2.9 / +1.7 / +1.6 / +1.4 / +1.1 / +0.6. The 512-sequence estimates were 3.7 / 3.0 / 1.8 / 1.7 / 1.4 / 1.1 / 0.5.

## What this means for the loss-attribution runs

- **The question to explain changes.** It is no longer just why register_a15 lowers loss. It becomes:
  - why wiki data lowers loss *more* than register data, while register scores better on benchmarks;
  - why α35 loses language-modelling gains on FineWeb, FreeLaw and Gutenberg while keeping them on domains that match the steering source.
- **Both questions fit what `loss_attribution_analysis/scripts` already does:**
  - the per-domain and contrast tables from `03_cross_arm`;
  - the bucket attribution from `02_attribution`;
  - the human-half checks (tasks 4 and 5 of `04_corpus_measures`), which separate a real α35 effect from the shift in human data.