Methods & references

§1How a sweep is measured

A sweep duplicates or prunes transformer blocks (no training) and asks whether capability moved. The whole difficulty is telling a real shift from seed noise, a generous grader, or the sheer number of configs tried. These are the rules that keep an answer honest.

Paired Cohen's d vs paired t

Every config is scored against an in-run baseline on the same seeds, so the test is the per-seed difference, not the marginal spread. d = mean(diffs) / std(diffs) is the seed-count-independent effect size; t ≈ d·√n grows with seed count. Real-but-underpowered candidates show high d, low t — worth more seeds, not yet proven.

The r-gauge

std(per-seed Δ) / std(baseline seed-noise). r < 1 → pairing cancels common-mode seed noise, so trust d even when the config's raw mean sits inside the baseline's spread (overlapping ranges are not "it's noise" — the paired diff is the test). r ≈ 1 → the difference noise is seed-independent; the config is riding the discovery noise floor and more seeds won't rescue it.

The multiplicity budget

A sweep of ~150–500 config×channel tests expects ~25 false positives at p<0.05 by chance alone. A lone t≈1.7–2.3 "discovery" is at that look-elsewhere floor, not above it. A result has to clear the budget — or be corroborated on another axis — before it counts.

Strict over partial

Strict grading (exact-equality, position-agnostic any-match on the answer integer) is primary. Partial credit — David Ng's pad-and-relative- difference scorer — rewards an answer for being close in magnitude, and it has manufactured phantom leaderboard champions on every model in this lab. It captures a real second axis (closeness) but is never folded into the headline. Writeups lead with strict + strict-correct %.

The grader-gaming taxonomy

Every model's first leaderboard is partly phantom, always the same way — a lenient grader rewards output form over correctness. Three faces: partial-credit inflation (close-but-wrong tidy numbers score ~0.99), echo contamination (the parser grabs the prompt's own numbers when the model just echoes), and compliance-not-capability (pruning unclogs a stalled output valve so the model finally emits a clean token — a real gain only if the latent knowledge was there). The fix is an operand-aware any-match strict regrade on the preserved verbose responses.

Seeds vs probes

Orthogonal. Seeds = precision (replication of the same battery); probes = coverage (breadth of what's tested). Bulking seeds on a thin battery makes it precisely narrow, not representative. Seed budget goes to the seed-independent channel — the one where pairing can't already cancel the noise.

Sampler & quant discipline

One fixed low-variance sampler per sweep (temp 0.3 + engine defaults, seed-controlled — greedy n=1 fabricates false-positives). A uniform sampler or quant level is a constant offset that cancels in seed-paired diffs; the only thing that can distort a ranking (versus an effect-size magnitude) is a sampler×config or quant×surgery interaction. So low-bit quant noise is real — it degrades the precision tails (exact arithmetic, tool-call tokens, long context) — but it is not a blocker for the broad-strokes capability a sweep measures. Magnitude of change is not sensitivity to change — the same distinction that separates a layer's geometric activity from its behavioural fragility.

Per-model specifics (quantisation, layer count, the per-layer-array reindexer, exact seed and probe counts) live in each specimen's own method note. This is the shared vocabulary, not the run log.

§2References

1. David Ng (dnhkng). RYS & RYS-II, the sapir-whorf and rys posts on dnhkng.github.io — the original "Repeat Your Self" technique, which he rode to the top of the open-LLM leaderboard. Stalled publicly on Gemma 4 31B over the per-layer-array problem; the reindexer used here solves that case. His 2026 retrospective ("never found a single duplication that generalised; could only boost one talent at another's expense") independently matches the archetype split found across this lab.
2. Maarten Grootendorst. A Visual Guide to Gemma 4. The shared-KV routing and the global/sliding attention pattern aren't documented in Google's model card; this guide is the reference.
3. HF transformers modeling_gemma4.py — the actual routing rule for shared_kv_states, read by hand to identify the sole attention anchors that make Gemma-4 surgery behave the way it does.
4. alainnothere. llm-circuit-finder, github.com/alainnothere/llm-circuit-finder. Independent RYS replication on Devstral-24B and Qwen2.5-32B — same layer_path.py primitive, different models, sharp circuit boundaries (incl. a +245% BBH-Logical-Deduction lift from a 3-layer duplication at L12–14). Proof the boundaries-are-sharp finding generalises off the Gemma-4 family.
5. Model authors — Google (Gemma), IBM (Granite), Sapient / Aryagm (HRM), and the respective authors of every other specimen, each under their own license. The architectures in question.

§3Build notes & credits

Back to the lab: the specimen index · Gemma 4 · Granite · HRM.