What this page establishes

• The surgery is trivial to do; the measurement is the whole contribution. Four distinct confounds (seed luck, lenient grading, multiplicity, exposure) each manufacture fake results, and each has a characteristic fingerprint you can subtract. The grading trap alone comes in three flavours: partial-credit leniency, answer echo, format-compliance-as-capability, dissected in §1.
• The seed is the unit of replication. No seeds means no paired test, no effect size, no confidence interval. That missing layer is exactly what published RYS work lacks.
• Pairing is what lets you trust a config whose scores overlap the baseline's: the per-seed difference is the test, not the spread, and that single idea reverses the most common wrong call in the field.
• Most of what people fear about local-model evals (quantization, sampler choice) cancels in that pairing. Only an interaction can distort a ranking, and you can test for one directly.

§0The one hard problem

What he doesn't report, and what nobody in published RYS practice reports, is seeds. Without seeds there's no replication unit: no paired test, no effect size, no confidence interval, no multiplicity correction. The reason is structural, not careless. The implicit model treats each generation as one independent draw from the truth, so why would you repeat it? The work below is the layer that assumption skips over.

It isn't a correction of his method. It's the statistics bolted onto it.

§1Phantom #1: the grader (every model's first leaderboard is partly fake)

Run any model's first sweep and you get a leaderboard. Some fraction of it is hallucinated, and the mechanism is always identical: a lenient grader scores output form (well-shaped, right-magnitude, compliant) instead of correctness. I've watched it manufacture champions on every model in the lab, in three distinct flavours.

Echo contamination

This one cost a full overnight. A naive integer parser grabs the last number in a response. When the model just echoes the question back, "What is 4,318 plus 2,901?", the parser scores partial credit on the input numbers. On Nanbeige, 66% of one overnight run was echo-only. Once you de-contaminate (blocklist each prompt's own integers), the real RYS effect grew: from +0.21 to +0.32. interpretation The baseline had been quietly scoring on echo too, so cleaning it up widened the gap. The signal was hiding under the baseline's own cheating.

Partial-credit inflation

A digit-overlap or magnitude-plausibility scorer gives a close-but-wrong tidy number ~0.99. On Gemma-E4B, combo_2_36_39 scored +6.11 on partial credit and −0.82 on strict (paired t vs baseline): a top-of-board "champion" that was, in fact, slightly worse than baseline once you demanded the right answer. A simpler config on the same model, prune_16, did the same, +3.45 on partial, a flat strict-zero: every "win" was magnitude-plausible garbage, not one exactly-right answer. The fix is a strict-equality regrade, and the diagnostic is the partial-minus-strict gap: sort the configs by that gap, and the biggest gappers are exactly your phantoms.

Compliance, not capability

Pruning a layer can unclog a stalled output valve: the model that was looping "Answer: Answer:" or stuck on a dangling colon finally emits one clean token. A partial grader reads that as a win. Whether it is a win depends entirely on whether the knowledge was already there: on multiple-choice factual recall, unclogging the valve surfaces a real answer the model knew (genuine gain); on 16-digit arithmetic it knew no answer to, unclogging just produces confident, well-formatted garbage (the strict phantom). Same mechanism, opposite verdict, decided by whether the capability was latent.

§2Phantom #2: the single draw (greedy n=1 fabricates results)

A champion scored 0/3 on the retest, worse than baseline, on a config an overnight sweep had crowned with a beautiful, theoretically-motivated story. That's the most seductive failure mode: the fake result is clean.

The lab has it written up as a cautionary tale: the HRM specimen. An overnight sweep, scored greedily and once per item, produced a gorgeous champion ("duplicate the renderer module and multi-hop reasoning improves"), corroborated by an elegant geometric metric. Then the verification pass sampled at temperature, three draws per item instead of one, and put the champion at 0/3, showing the base model already solved the puzzle ~70% of the time on its own. observation The greedy sweep had been measuring which single draw got lucky, not capability. The corroborating geometry turned out to be decoupled from the outcome it seemed to confirm.

The rule that fell out is absolute: greedy / temperature-0 / n=1 fabricates false positives on this entire research line. HRM, Gemma, and Nanbeige were all bitten. Use a fixed, low-variance sampling sampler (temp ~0.3 plus engine defaults), seed-controlled, and average over many seeds. Greedy is fine for characterizing a baseline, where there's no knife-edge config comparison, and nowhere else. This is why the seed is the unit of replication, and why everything statistical below is built on seeds. No seeds, no test.

§3Pairing: why "the ranges overlap" is not "it's noise"

The most common way a careful person still gets it wrong: you compute a config's mean score, the baseline's mean score, you notice their ranges overlap, and you conclude the difference is noise. That conclusion is unjustified, and seeing why is the whole game.

Every config is scored against the baseline on the same seeds. Think of each seed as a fixed exam paper handed to both students. If seed 7 is a hard paper, it's hard for the config and the baseline — so the gap between them on seed 7 has that shared difficulty subtracted out. The marginal spread of either score is dominated by which papers got drawn; the thing you actually care about is how the config did relative to baseline on the same paper:

paired Cohen's d  =  mean(per-seed diffs) / std(per-seed diffs)

d is seed-count-independent: an effect size, not a test statistic. The test statistic t ≈ d·√n grows with the number of seeds; d doesn't. So a real-but-underpowered config shows up as high d, low t (worth spending more seeds on, not yet proven), and you rank your retest queue by exactly that signature. A second number tells you whether to trust d over the eyeball test: the ratio of the paired-difference noise to the baseline's own seed noise:

r  =  std(per-seed Δ) / std(baseline seed-noise)

• r < 1 → pairing is cancelling common-mode seed noise. Trust d even when the config's raw mean sits inside the baseline's spread. The overlapping ranges are not the test; the paired diff is. This is the case people misread as "noise."
• r ≳ 1.4 (≈√2) → the difference noise is seed-independent; pairing buys you nothing and more seeds won't rescue it. This config is riding the discovery noise floor.

And this gives a budgeting rule that's the opposite of the naive one. You don't spend seeds where the variance is highest: you spend them on the seed-independent channel, the high-r one, because that's where pairing can't already cancel the noise for you. Empirically, no-CoT math is the high-r channel (so it eats the seed budget and needs sign-consistency checks, since a high-d config with scattered diff signs is a magnitude mirage, drop it); multiple-choice and EQ are low-r. interpretation Seeds are precision; probes are coverage; they're orthogonal, and bulking seeds on a thin battery just makes it precisely narrow.

§4Phantom #3: the multiplicity budget

A sweep is a multiple-comparisons machine. The intuition is the lottery: buy 300 tickets and one of them winning tells you nothing about your skill at picking numbers, somebody had to win. Run ~150–500 config×channel tests and you should expect ~25 false positives at p<0.05 by chance alone. So a lone discovery sitting at t ≈ 1.7–2.3 is not a finding: it's exactly at the look-elsewhere floor, indistinguishable from the noise you expect a sweep that size to throw up.

A result clears the bar only if it (a) beats the multiplicity budget outright, or (b) is corroborated on an independent axis — a second channel, a second seed batch, a mechanistic prediction that came true. A single t in the 2s, standing alone, is a candidate, not a conclusion. The discipline is cheap to state and expensive to hold to, because the seductive result is always the lone one: the config nothing else corroborates is the config you most want to be real.

This is why the lab's structure is two-tier. A cheap, narrow battery (the no-CoT delta-detector, 2–3 seeds) ranks all configs — explicitly not ground truth, just an intuition delta-detector. Then a richer, slower eval (more probes plus a real generation task, ≥5 seeds, with the wide-variance configs re-run) runs on the winner shortlist only. Round-1 sorts; round-2 confirms. Never make a public claim off the thin battery. (xpressai learned this the hard way and said so: their small internal probe reported a RYS config had lower EQ; their larger held-out probe said higher. "Small-probe variance was misleading us." Same lesson, paid for independently.)

§5Read the baseline before you trust any config

First, profile what the baseline leaves room to measure

A channel can only measure what its baseline leaves room to measure, so you profile the baseline's per-channel saturation and per-seed variance first. Picture each channel as a ruler with a usable middle and two dead ends: push the baseline against either end and there's no room for a config to register. The recurring profile across dense models at aggressive quant:

• math-strict is floored — huge no-CoT arithmetic is impossible, the column is near-dead, zero discriminative power. Only easy math (baseline 0.2–0.8) gives a live strict signal.
• multiple-choice is ceilinged — bare-letter MMLU sits near 1.0, so it's a downside-only damage detector. It maps which layers are load-bearing for retrieval; it structurally can't show an improvement.
• EQ is the bidirectional channel, with real two-sided headroom, but often the dirtiest. On Gemma-31B the EQ standard deviation was 0.277, per-seed values swinging 0.00↔0.80; every EQ "leader" sat inside baseline noise. The bimodality turned out to be a missing answer-priming suffix ("\n<emotion>:"), not signal. A channel's t-stats are noise until you understand its variance.

A related subtlety: no-CoT math is not binary. On Gemma-31B (n=4896), strict was 0.44 and partial 0.76: 43.6% both-correct, 48.7% a smooth near-miss continuum, only 7.7% outright junk. The partial scorer is capturing a real second axis, magnitude/shape closeness, so a config can tighten sub-threshold near-misses without moving strict at all. That's why partial stays an annotation on the strict leaders, never folded into the headline: it's real information about a different capability, and it's also the exact thing that inflates phantoms. Both, at once.

Then calibrate the channel before you spend compute

The usable middle of the ruler is 40–70% baseline pass. Above ~80% is ceilinged (no variance to move); below ~20% is floored (can't detect improvement). If a channel is out of band, recenter the probes (smaller operands, not new problem types) to land the baseline mid-band before you reseed. Reseeding a ceilinged or floored or bimodal channel just measures the artifact at higher resolution: wasted compute. The sequencing rule, in order: fix the probe / grader / suffix first, reseed second.

§6The metrics that earned their place across models

Four measurement habits generalized off specific models and are now run by default.

Degeneration rate (from MiniCPM5 and Granite): a zero-compute count over already-stored responses, how often a generation derails (special-token leaks (<|fim_prefix|>), name:number line-loops, digit repetition). It's monotone in both surgery knobs (copy count and cumulative-α), and it turns out to be the real cost axis. Throw out only the derailed responses and score the clean ones, and the surgery improves content quality on every channel: the only thing it costs is derailment probability. That reframes a lot of apparent "EQ craters" as a single mechanism, channel divergence ≈ exposure divergence. A short channel (~16-token math) barely gives a low per-token derailment risk room to fire; a long channel (~96-token EQ) rolls the dice five times as often. The "EQ crater" isn't lost competence: it's the same small risk, exposed longer. Put a degen-% column on every report.

The fine-α grid is mandatory; the 3-point map lies. A prune(α=0)/baseline(α=1)/dup(α=2) skeleton is invisible to any layer with a U-shaped α-response — Granite's L6 peaks at α≈0.7 while both prune and dup collapse it. Any per-layer sensitivity claim needs the dense single-pass α grid (~0.5–1.5), not the dup/prune endpoints.

The other two are ranking hygiene, stated briefly. Rank by balanced-t, and treat no t as a p-value. The harness "strict t" pools every (probe×seed) cell, over-weighting whichever channel has the most cells; the monitor "balanced t" gives one vote per channel. Rank by the latter, and read both as ordering devices only.

Per-probe autopsy is a lens, never the rank target. A balanced 49-probe average dilutes concentrated capability flips, and often that's the real win: Granite's deterministic borrow/carry errors were wrong the same way on every seed, and the winning config flipped them 12 of 12. Read per-probe to see which bug got fixed; never rank by max-over-probes, which just rewards a lucky single tip.

§7What cancels, and what doesn't

A recurring worry on local-model work is that aggressive quantization or sampler choice contaminates everything. The seed-paired design dissolves most of it, and the precise version tells you exactly which worry survives.

A uniform sampler setting, a uniform quant level, even the engine's own nondeterminism: each is a constant offset that applies equally to the config and the baseline on a fixed seed, so it cancels in the paired difference. Same exam paper, same handicap on both students. The only thing that can distort a ranking (as opposed to an effect-size magnitude) is an interaction: a sampler×config or quant×surgery effect, where the offset differs across configs. And a baseline-only ladder structurally cannot detect an interaction: you need ≥2 configs across the ladder to see one.

So the scope statement is: quant noise is real, and it degrades the precision tails (exact multi-digit arithmetic, tool-call and structured tokens, long-context, strict instruction-following), which is exactly where production agentic use breaks even when broad-strokes sweep capability doesn't. But it is not a blocker for what a sweep measures, and it never touches rank or direction. The clean way to settle this rather than argue it is a 3-point quant ladder (IQ2 / IQ4_XS / Q8) on baseline plus a few standouts, reading the shape of per-config Δ across the three: flat = pure offset, monotone = graded but rank-preserving, a threshold flip = the worry vindicated and the cliff located.

§8Coda: belief versus behaviour

Caveats

This methodology has held up across every specimen in the lab, and produces usable results once a new model is tuned in. The statistics are the primary addition here; Ng's probe design deserves full credit for the eval design this builds on.

• The layer-damping surgery may be a novel extension of the RYS method, not just the statistics. As far as I know, RYS and similar work hasn't attempted this kind of damping approach elsewhere — that's a separate, unconfirmed claim from the measurement methodology this page focuses on.
• No t is a p-value here. Every t-statistic in this lab is an ordering device read against a multiplicity budget, not a significance test reported in isolation.