Every other specimen in this lab gave up a winning surgery — a layer you could duplicate for a real gain. Gemma-4‑31B gave up nothing. But "nothing" turned out to have a shape: this model is robust to a single cut and rigid under a structural one, and that contrast is the finding.
gemma-4-31B (dense), the prestige target of the Gemma line: sixty blocks in a [5 sliding + 1 global]×10 rhythm, no mixture-of-experts, no shared-KV anchor. Reputationally a conversational/creative model (a spiritual successor to the warm, fluent GPT-4o register), which turns out to be exactly the problem with testing it on arithmetic. Hold that thought; it's §6.
Run on consumer hardware: the base checkpoint (not instruction-tuned — that choice is load-bearing, see §6) at IQ2_M, the heaviest quant that fits a 16 GB Mac. The sweep is a frankenstein factory: rebuild the GGUF with layers duplicated or deleted, reload, re-measure. No training, ever.
- Layers
- 60 · dense
- Rhythm
- 5 SWA : 1 global
- Globals
- L5,11,…,59
- Attn
- use_alt (V=K)
- Anchor
- none
- Engine
- llama-server
- Quant
- IQ2_M (base)
- Surgery
- rebuild GGUF
layer_path.py - Throughput
- ~6.4 tps
§1Robust to a cut, rigid under a structure
Duplicate almost any single one of Gemma-31B's sixty layers, or delete it, and the model barely flinches. The exceptions are structural and few: the embedding-adjacent pair (L0–1), the final pair (L58–59), and a couple of mid hinges (L35, L40) zero out completely. But across the entire interior of both ladders, no single-layer surgery moves capability outside the seed-noise envelope. Read alone, that reads as robustness.
But robustness is the wrong word, and the blocks prove it. Chain a few layers into a duplicated block and the model doesn't degrade gracefully — it craters, and it craters faster than the sum of its parts would predict from averaging. The damage isn't additive; the layers interact. Each block wants its exact neighbours, in its exact order. That isn't a robust stack. It's a rigid, positionally specialised one that happens to absorb a single perturbation.
The contrast that makes this legible is Granite and the small dense models before it — nanbeige especially, where you could lift a layer out of one depth, drop it somewhere completely different, and the model would sail straight through. Those layers were generalists: position-independent, transplantable. Gemma-31B is the opposite animal. A single layer survives being copied because the residual stream routes around it; a block of layers cannot be copied because you've changed the order the model is built on.
The negative result isn't "nothing happened." It's that this model is held together by where its layers sit — and a sweep that only moves one layer at a time can't see the glue.
Perturbation-robust — capability is invariant to a single dup or prune. Gemma-31B has this.
Position-robust — a layer keeps working after you move it. nanbeige had this; Gemma-31B does not. The block dose-response below is the measurement of its absence.
§2The winners that aren't
Three configs sit a little above baseline on strict math. The discipline of this lab is to not believe them. The gauge that matters is r: the standard deviation of the per-seed paired difference, divided by the standard deviation of baseline's own seed noise. When r<1, pairing cancels common-mode noise and the effect is trustworthy. When r≈1, the difference is just more noise, and seeds won't save it.
| config | math strict | Δ vs base | d | t | r | verdict |
|---|---|---|---|---|---|---|
| baseline | 0.736 | — | — | — | — | 12 seeds |
| dup_22 | 0.806 | +0.069 | +0.51 | +1.8 | 1.16 | noise floor · n=12 |
| prune_14 | 0.792 | +0.073 | +0.83 | +2.3 | 0.64 | above floor · n=8 |
| dup_9 | 0.792 | +0.073 | +0.75 | +2.1 | 0.71 | above floor · n=8 |
| prune_2_14 | 0.781 | +0.062 | +0.37 | +1.0 | 1.24 | noise floor · n=8 |
Two configs clear the floor, at half budget. prune_14 (r=0.64) and dup_9 (r=0.71) dip under 1 with d≈0.8, while the only fully-seeded leader, dup_22, sits at r=1.16. But both sub-1 configs ran just n=8 seeds: the sweep was killed before their budget filled, and a shallower seed count flatters r. Even at face value, t≈2.3 is inside the look-elsewhere budget: ~176 configs × 3 channels ≈ 500 tests, ~25 false positives at p<0.05 expected by chance alone. And what actually sinks them is §7: balanced across all three channels, the math gain evaporates to net ≈ 0. There is no positive result on the math axis. That is an honest answer, not a failed one.
§3Block damage: span × depth, floored
If single layers don't move the model, the obvious next question is whether a contiguous block of "good" layers, duplicated together, surfaces a reasoning circuit (the move that paid off on the dense E4B). It does not. It only ever costs. And the cost has a clean law.
Fig 1 — Block dup math-strict vs depth. Faint dots = single-layer dups (the "parts"), which hug baseline everywhere. Terracotta discs = duplicated blocks, radius ∝ span (2→5+ layers). Two gradients are visible: blocks fall as they get bigger (radius) and as they sit deeper (rightward). A 3-layer block at L2–4 costs −0.15; the same 3 layers at L37–39 hits the floor (0.000). Depth and span both bite; nothing recovers baseline.
Worse than the average, floored below the sum
Take a block's constituent single-layer dups and ask what the block "should" score. Two predictions bracket it. The average of the parts is the optimistic one — and blocks land far below it (blk_12_16: actual −0.57 vs part-average −0.16). Damage compounds beyond averaging: the layers interfere, they don't just add.
The sum of the parts is the pessimistic one — and blocks land above it (blk_2_6: actual −0.34 vs part-sum −1.01). That's not mercy; it's the floor. Strict score can't go below zero, so once a block bottoms out, extra bad layers have nothing left to subtract. Compounding, then saturation.
The combos say the same thing, quieter
Cross-depth combos (two single dups at different depths) are the gentlest multi-layer move and the only ones that stay near baseline: combo_15_22 (+0.07, t=0.86), prune_2_14 (+0.06, t=1.29) — the one structurally tidy config, holding mc and eq at baseline too. But none beats a single dup, and the moment a combo spans a real interval it joins the cascade.
The intended payoff (the E4B trick of chaining near-baseline layers to bracket a reasoning circuit) finds no circuit here. There is nothing to amplify; there is only structure to damage. The deepest blocks take EQ to 0.00 outright: not wrong answers, generation-mode collapse.
§4The noise lives at the front
A prediction we made early (and held too firmly), from the very first geometry scan, where the late layers rewrite the hidden state by far the hardest: later layers should be the fragile ones, noisier and less trustworthy under surgery. On Gemma-31B's behaviour, that is simply false, and the full ladder says so in both directions. The scan wasn't wrong; we read magnitude as fragility (see the walkback below).
Fig 2 — Per-layer seed noise (std of per-seed math means), L0–59. Terracotta = dup ladder, slate = prune ladder; dead/bricked configs (zero-variance constant garbage) excluded so they don't masquerade as "stable." Both ladders agree: the front third is the noisiest, the mid stack is calmest, the late third sits in between (the opposite of the quant-noise prediction).
Zone-averaged over both ladders: front L0–19 = 0.165, mid L20–39 = 0.105, late L40–59 = 0.120. The noisiest individual configs are all up front — prune_19 0.29, dup_7 0.28, prune_4 0.28. The prediction that duplicating a heavily-quantised late layer would propagate 2-bit noise is firmly refuted: late targets are quieter, on both surgery types.
The reconciliation is a distinction worth keeping: magnitude of change is not sensitivity to change. The late layers genuinely rewrite the hidden-state geometry the hardest (the scan-delta measurement, Fig 3), but that's deterministic output-shaping, the detokenisation regime a logit lens predicts, not noise-amplifying computation. The front layers carry the computational load; a late perturbation washes out before it reaches the logits. The two axes decouple with depth. Late layers are absent from the leaderboards not because noise buries their signal, but because their effect is null — or they brick.
Bricks, not noise. The global-grid catastrophics (dup_0/1, late globals) skew slightly toward the back — but a brick is zero-variance constant garbage, categorically different from seed scatter. Degeneracy ≠ instability.
Fig 3 — Geometry change vs behavioural noise, by depth. Wine = per-layer Δ/residual ratio (how hard each layer rewrites the stream, 2-bit scan-delta; boundary layers L0 & L59 spike off-scale to 1.5 and 13.5). Terracotta = per-layer seed-noise (Fig 2, right axis). Opposite slopes. The geometry rewrites hardest exactly where surgery does the least behavioural damage. Magnitude of change is not sensitivity to change.
We expected the late layers to be the fragile ones. The very first thing we ran on this model was a 2-bit hidden-state scan, and it showed the deep layers rewriting the residual stream far harder than the front. We read that as fragility: "duplicate a heavily-quantised late layer and its 2-bit noise will propagate," and the assumption quietly coloured how we read everything after. Then the full dup and prune ladders came in and said the opposite: the front is the noisy zone, the late stack is calm. The scan was never wrong. We had mistaken the amount a layer changes the stream for how sensitive the output is to changing that layer; those are different measurements. The late layers move the stream a lot because that's where token-shaping happens; perturbing that shaping is deterministic, not noisy. Naming the prior is the point: it's the kind of assumption that survives precisely because it sounds mechanistic.
§5What the partial grader compresses
Every page in this lab carries a warning about partial-credit math grading: David Ng's elegant pad-and-relative-difference scorer, which rewards an answer for being close in magnitude. On the 26B MoE it manufactured a fake winner: a config that gamed magnitude rocketed up the partial leaderboard while strict saw nothing. The lesson there was rank-inversion.
On 31B the grader misbehaves differently, and more quietly. Strict and partial correlate at r=0.85: no rank-inversion this time. But partial sits a flat +0.36 above strict and saturates: baseline already scores 0.96 partial, and nearly every config piles up between 0.92 and 1.0. Partial can't tell the top configs apart — they're all ceilinged. It only moves when a block craters hard enough to stop emitting plausible numbers at all.
So the failure mode flips: on the MoE, partial lied about the ordering; on 31B, partial erases the ordering at the top. Either way the conclusion is identical — strict is the only axis you may headline. The any-match strict grader (correct integer present anywhere in the output, operands filtered) is position-robust by construction, so a strict zero is a genuine miss, not a formatting artifact.
§6What this negative does — and doesn't — cover
The honest scope of the result is narrower than "RYS does nothing to Gemma." Three boundaries matter, and naming them is what separates a clean negative from an overclaim.
We may be testing the wrong organ
Gemma-4 is, by reputation, a conversational and creative-writing model (the warm GPT-4o register, not a math engine). Our battery is arithmetic, factual recall, and formal logic. If the layers that are worth duplicating are creative-writing circuits, a math/logic sweep would see exactly this null. There is no cheap objective grader for "complete the simile" — so the creative axis stays untested, and the result is scoped to reasoning/recall, not universal.
The closest probe we have to that native register is eq-sketch: open-ended equation completion. We kept it at three seeds and chose not to deepen it: it's the most expensive channel to run and the most stable, so extra seeds buy almost no variance reduction. The budget went to finishing the sweep instead. And the right way to reach the creative axis isn't more eq on a 2-bit 31B — it's the sibling Gemma-4-12B: same family, near-identical block structure, small enough to run at a real quant. Findings here should carry over, and the muzzle lifts. That's the next target, not a deeper 31B re-run.
The base + no-CoT muzzle
We test the base checkpoint in no-chain-of-thought, base-completion mode — answer in a handful of tokens, no room to think. That is deliberate (the instruction-tuned model fights the format; base blurts a number), but it drags the model out of its native zone and pins the strict ceiling. Pending control: confirm the ceiling is the muzzle by letting base reason (CoT) and by running a winner on the IT model with room to respond. The verdict shouldn't change; the framing earns its evidence.
Everything here runs at IQ2_M, the quality floor that a 16 GB RAM ceiling forces. The geometry scans show 2-bit heavily reshaping the deep-layer hidden state (Fig 3), so a late-layer positive could in principle hide under the quant ceiling (the behavioural noise map argues against it, see the §4 walkback, but can't prove a negative). The standard fix, sweeping the same model at 4-bit, doesn't fit the hardware.
The hardware-bounded version that does fit: build a mixed-precision GGUF (bump only the RYS-target and late layers to Q4/Q5 while the rest stays IQ2) and re-sweep just those. It isolates "is the late-layer null a quant artifact" without ever loading a full 4-bit 31B. That is the concrete replication path, and it's the right next experiment for anyone who wants to push past the consumer floor.
§7Preliminary candidates
With no statistically defensible winner, these are offered as preliminary: the configs that sat highest on strict math, for others with more hardware to confirm or refute. Treat them as "try these," not "use these."
And the cross-channel aggregate doesn't endorse them either. The strict-math gain is the only place these configs win. Balance all three channels equally and it evaporates: dup_22 −0.037, prune_14 +0.006, dup_9 −0.002 — net zero to negative. The math bump is paid for on eq-sketch and MC. (eq-sketch ran only n=3 seeds, a deliberate budget call, not a truncation: it's the most expensive channel and the most stable, so extra seeds buy almost nothing. The negative eq Δ is therefore the trustworthy part; only its Cohen's d is wide at n=3.) These aren't suppressed winners; they're noise-floor configs with one flattering channel.
Publishing these inherits Google's Gemma Terms of Use, not Apache — the license, "Gemma" attribution, and use-restrictions travel with any derivative. Credit chain: Google (Gemma) → David Ng (RYS technique) → this lab.
§8Notes from the rig
- Sliding vs global is not a behavioural distinction here. Every probe runs well under the sliding-window length, so a sliding layer attends to the entire sequence: the attention math is identical to a global layer's. The catastrophes at the global grid (L5,11,…) are structural: the V=K tie of use_alt attention and the per-layer-array reindex (sliding_window_pattern, head_count_kv), not a receptive-field effect. The boundaries matter for tensor shape, not for what the layer can see.
- Seeds, per channel. 12 for math/mc baseline, 8 for the slow eq channel; 8–12 for candidate configs. Paired-difference stats match seeds on the intersection, so a config is only as deep as its shallower half — the baseline was deepened in lockstep so the pairing never wastes a seed.
- Base-vs-IT is settled at the substrate level. Base is the right RYS substrate: the IT model restates the equation or hedges before answering, base blurts a number first; base out-scores IT on eq and mc once priming lands. The dramatic base-vs-IT geometry difference decomposed to quant (3→2-bit), not tuning. What remains pending is carrying a winner onto IT with room to respond.
- The receipt. ~32 GPU-hours, ~34k probe-cells (reruns from growing the battery included), ~418k generated tokens, ≈1 kWh. The write line is the surprising one: the GGUF is mmap-read at inference, but every config rebuilt the frankenstein twice: the layer-surgery copy, then a second copy with the per-layer-array fix, ~10 GB apiece, both deleted after. Across 176 configs plus reruns and reseeds that's several terabytes shuffled through the NVMe — real, but ~1% of the drive's rated endurance. A load of laundry's worth of electricity to prove a model is stubborn.
Scripts: gemma_31b_v2_probe_test.py · shared/probes_combined_v2.py (any-match math + primed eq + balanced mc) · gemma_fix_per_layer_arrays_31b.py (SWP + head_count_kv reindex) · monitor.py --sweep 31b_v2. Data: gemma_31b_v2_base_results.jsonl (per-cell, 176 configs × per-channel seeds). Sister specimens: Gemma 4 E4B / 26B · Granite · geometry in the appendix. Shared methods & references.