Gemma 4,
opened up at every layer.
A 42-layer language model — Gemma 4 E4B — is duplicated, pruned, swapped and recombined one residual block at a time. Most of these surgeries make it worse. A few make it better, and the pattern of which-is-which turns out to be almost entirely explained by where the model stashes its keys and values. Then the same scalpel meets a larger, stranger patient — the 26B-A4B mixture-of-experts — which refuses to respond the way the dense sibling does. Why it refuses is the most useful thing the bigger specimen has to teach.
[5 sliding + 1 global] × 7
swept end-to-end
L22 sliding · L23 global
L24 – L41
strict-correct rate
On April 11, 2026, a discussion thread opened on the Gemma 4 31B model card: could someone with the hardware run David Ng's RYS layer-duplication technique on it? Ng, who created RYS and rode it to the top of the open LLM leaderboard, estimated about 40 H100-hours and said he'd get to it when compute allowed. The thread went quiet; the 31B never got its surgery. (Ng has since returned to RYS on the current model generation. See his 2026 retrospective, linked below.)
Meanwhile, across April and May, this experiment ran on a Mac Mini M4 with 16 GB of unified memory, sweeping 247 surgical configurations on the smaller E4B sibling. It is the first known working RYS sweep on the Gemma 4 family, unblocked by reverse-engineering three undocumented per-layer metadata arrays that the upstream tooling silently leaves stale after a duplication. The original technique was for somebody with H100s. This one was for somebody with a kitchen.
- Hardware
- Mac mini M4 (2024)
- Memory
- 16 GB unified
- Storage
- 245 GB, 43 GB free
- OS
- macOS Tahoe 26.3.1
- Retail
- ~ $599
- Hardware
- NVIDIA GH200 Grace Hopper Superchip (per his benchmarking post)
- Memory
- ~ 624 GB unified (LPDDR5X + HBM3)
- Compute
- H100-class GPU, 72-core Grace CPU
- OS
- Linux, CUDA
- Retail
- ~ $40,000+
§1The patient
Gemma 4 E4B is a small open-weights model published by Google, with one architectural feature that turns out to dominate everything else for this work: it does not actually carry its own keys and values in the back half of the network. Layers 0–22 are writers that compute their own Q/K/V projections. Layers 24–41 are consumers: at inference time the engine routes them to read from a two-slot dictionary written by exactly two layers: L22 (the sole sliding-attention anchor) and L23 (the sole global-attention anchor). The 18 consumers all read the same frozen cache. Their own K/V weights sit on disk and never fire.
This means there are really three Gemmas inside the same checkpoint. A writer zone that does most of the heavy encoding. Two anchors whose output is broadcast to every consumer downstream. And a long consumer chorus that refines what the anchors already decided. The layer strip below colours each layer by which of these it is. Hover for the details.
§2What happens when you touch a layer
Two surgeries, applied one layer at a time. Dup: duplicate the layer so the residual stream passes through it twice. Prune: delete the layer and stitch the network back together. Each bar is a paired t-statistic against the 15-seed baseline on a 16-probe math battery. Above zero, the patient improved. Below zero, the patient got worse. The wine bar in the middle is L22 — the sole sliding anchor. It is not touchable; both surgeries crater on it.
The crater gradient
Layers 18–22 form a smooth crater approaching L22 from below. The further the duplicated layer is from the anchor, the less the damage propagates undamped. By L2 (twenty layers of damping away), the delta-norm has attenuated enough that amplification flips from catastrophic to beneficial.
The prune-rescue asymmetry
Several layers (L3, L16, L18, L36) are positively helpful to remove. Not just "less bad than duplicating", but a real gain over baseline. That makes pruning an independent intervention axis, not the inverse of duplication. Some layers are noise.
§3The grader was lying
The original sweep used a partial-credit scorer borrowed from the upstream RYS work — digit-overlap, right-zero-padded. It gives a 17-digit answer that's off by two digits in the middle almost full credit. When the leaderboard was re-run under strict equality, half the "champions" turned out to be writing beautifully-formatted close-but-wrong numbers — a model that looks like it can do arithmetic without actually doing it.
§4What survives strict scoring
Once the partial-credit gaming is stripped out, the real lift is smaller but still substantial: from a 2.9% baseline correct-rate to ~10% in the best configurations. That's a 3.5× improvement in actual correctness from rearranging layers, with no training and no fine-tuning.
The survivors fall into three honest patterns. Single-layer dup at L2 (small per-layer delta, twenty layers of damping before reaching the anchor). Pruning at L3 (a noise contributor in the encoder). And the late consumer-zone wins, L39 and L40 plus blocks and combos that stack them, which exploit the shared-KV chorus by consulting the frozen anchor cache one more time with a refined residual.
Real winners (strict_t, n=247)
| Config | Operation | strict t | correct % | partial t | Pattern |
|---|---|---|---|---|---|
| block_39_40 | consumer-zone block | +1.75 | 10.4% | +4.09 | highest strict-correct rate |
| prune_3 | writer-zone prune | +1.62 | 10.4% | +2.05 | noise-contributor removal |
| combo_2_36_39_40 | cross-zone (L2 + 3 consumers) | +1.92 | 9.4% | +2.98 | new champion · 4-layer stack |
| combo_2_4 | writer-zone dup pair | +1.39 | 9.4% | +0.38 | upgraded under strict |
| dup_40 | consumer-zone single dup | +1.42 | 8.3% | +1.32 | pos-4 in last block |
| dup_2 | writer-zone single dup | +1.80 | 6.3% | +3.55 | the foundational win |
| combo_2_39 | cross-zone (L2 + L39) | +1.00 | 7.3% | +4.36 | demoted from partial-t #2 |
| dup_5 | writer-zone single dup | +0.38 | 5.2% | -0.16 | qualitative-clean win |
The catastrophic floor
| Config | Operation | strict t | correct % | null % | Failure mode |
|---|---|---|---|---|---|
| dup_0 | embedding dup | -2.68 | 0.0% | 80.0% | "Answer:" token loop |
| dup_1 | second-encoder dup | -2.68 | 0.0% | 40.4% | question-echo loop |
| dup_8 | writer-zone dup | -2.68 | 0.0% | 39.2% | circuit disruption |
| dup_21 | anchor-adjacent dup | -2.68 | 0.0% | 15.0% | cache corruption — undamped |
| dup_22 | SOLE SLIDING ANCHOR | -2.68 | 0.0% | 1.7% | HTTP 500 — server crash |
| dup_41 | final-layer dup | -2.03 | 0.0% | 94.8% | effectively a refusal |
| prune_40 | consumer pos-4 prune | -2.03 | 0.0% | 90.6% | severs late-refinement path |
| prune_24 | consumer pos-0 prune | -2.03 | 0.0% | 78.1% | severs writer→consumer handoff |
§5Failure modes, by layer
Reading the verbose responses by hand, the way each layer dies is distinct — not just gradations of "less correct". Each surgery selectively breaks something different in the model's machinery. A few specimens, one per cluster:
§6Where the work actually happens
A side experiment, separate from the sweep: trace per-layer hidden-state deltas on a held-out set, stratified by the model's prediction confidence. The hypothesis was that the consumer chorus does more work on uncertain prompts. It does not. The shared-KV zone is flat regardless of input difficulty, with a 0.4% gap between high- and low-confidence prompts. There is no adaptive-compute mechanism. The model thinks the same amount about everything.
The clean two-zone story above is the writeup. The actual process was messier. Hypotheses that did not survive contact with the sweep, in order of disappointment:
- Distance-to-anchor as predictor. Initial framing was that writer-zone t-stats are linear in distance from L22. Pearson r in the writer zone came in at +0.14; consumer zone at −0.03. Quietly retired.
- Position-within-block as a universal pattern. Two pos-0 winners (L30 = +2.78, L36 = +2.73) lined up perfectly across blocks. The prediction was that L40 (pos-4) would land around −1. It came in at +1.59. The modular structure exists as a secondary signal, not a deterministic one.
- "Shared-KV consumers are un-RYS-able." Earlier theoretical claim grounded in the routing mechanism. Falsified by dup_30 = +2.78 within the first dozen consumer-zone configs. Replaced by the refinement-amplification story.
- combo_2_36_39 as champion. Held the top of the partial-credit leaderboard at +7.41 for several days. Strict regrade dropped it to 2.1% correct — below the 2.9% baseline. The actual champion (combo_2_36_39_40) was further down the original list.
- prune_16 as the best single prune. Same shape, same regrade outcome — 0% strict-correct. The pattern was "L16 pruning produces null or near-null outputs that the partial grader treats as digit-overlap successes."
At one point the progress counter wrapped: Configs done: 247/37 (668%). The sweep had been planned for thirty-seven configurations and ended at two hundred and forty-seven. None of the additional two hundred and ten changed the final mechanism story — but four of the five interpretations above were retracted in the process of running them.
§7Two zones, two mechanisms
The headline contribution isn't a number on a leaderboard: the absolute correct-rate is still 10%. It's the picture of why layer surgery works at all on a model with such a tight K/V bottleneck, and where the room is.
In the writer zone (L0–22), the rule is delta-norm × distance-to-anchor. A duplicated layer is useful only if its perturbation is small enough to survive twenty layers of damping before reaching L22. L2 is the only singleton that fits.
In the consumer zone (L24–41), the rule is different: every consumer reads the same frozen anchor cache, so duplicating one isn't a perturbation, it's an extra refinement pass on top of unchanged state. Gentle, often helpful, occasionally a real win.
Cross-zone combinations, writer-zone L2 stacked with consumer-zone L36, L39, L40, produce the only superadditive results in the sweep. That's the load-bearing publishable finding. Single-zone amplification is bounded by the anchor; the room for actual lift is in the interaction between the two mechanisms.
Two things are missing from this picture, both honest. dup_23, the sole global anchor, was attempted — llama.cpp asserts on map_layer_ids and the model never loads, so it returns no data points, only a stack trace. And the gap between partial-credit and strict scoring is a methodological wound the whole RYS literature should probably take seriously. The original score was overcounting close- but-wrong by 3–4×.
§8Postscript: different probes, different layers
Sweep complete · 183 configs · three archetypes identifiedThe mathematical correctness sweep selected for one capability. A second sweep — 4 emotion-attribution scenarios, 10 multiple-choice general knowledge items, 2 sanity math probes — produces a meaningfully different layer-sensitivity map. Across 183 configurations covering the same 42-layer model under two objective surfaces, three distinct surgical archetypes emerge.
Math specialists
Big lift on arithmetic intuition; factual recall and emotion attribution take a hit. The model gets better at one thing by getting worse at others.
EQ specialists
Large emotion-attribution gains; math and sometimes factual recall pay for it. The most amplifiable axis in the project lives here.
True generalists
Math+ AND EQ+ AND factual recall preserved. The rarest outcome — most surgeries trade between capabilities, these don't.
§9The larger specimen: a mixture-of-experts
Two sweeps · math + domain · compliance, not capabilityEverything above is Gemma-4-E4B — four billion parameters, forty-two layers, a tight KV bottleneck. This is a first look at its larger sibling: Gemma 4 26B A4B, the instruct-tuned variant, scanned via gemma-4-26B-A4B-it-heretic-msq-2.6bit. The scans came first — a look at its internal geometry before any cutting. The surgery came after, and it's the main event below.
Two flags on the specific weights used here. msq-2.6bit is an aggressive mixed-precision quantization: necessary to fit the 26B into MLX on a 16 GB M4 Mac Mini at all. Whether that level of compression meaningfully distorts the internal representations being measured is not fully known; treat the scans as indicative rather than definitive.
heretic designates an ablated-refusal fine-tune. The scan was run on this checkpoint simply because it was what was on the drive. Refusal ablation is unlikely to dramatically alter the mid-network semantic geometry being measured here (it operates primarily on late-layer output behavior), but it is a confounding factor that should be controlled for before any surgical conclusions are drawn.
Getting per-layer hidden states out of an MLX model turned out to be non-trivial. Every existing writeup on extracting intermediate activations assumes a bf16 model running under HuggingFace Transformers — hooks, output_hidden_states=True, done. None of that applies here.
The actual solution required patching the MLX model's forward pass to return intermediate layer outputs, then wiring those through the mlx-lm generation loop without breaking the sampling machinery. The details were worked out in a separate Sonnet session while the human half of this project was fighting with virtual environments and Python path issues. That session gets the credit. This one just ran the numbers.
Part of the point of the scans was simply to find out whether this hardware could load and probe a 26B mixture-of-experts (and a 60-layer 31B dense) at all. It can, at aggressive quantization. What the probing found, before any cutting:
On the 26B, one signal survives the quantization noise: math. The EN↔ZH math pairs begin maximally dissimilar at the embedding, then climb into positive territory through the middle third of the network (mathematical relationships compressing into a more language-agnostic representation) while factual and poetic prose stay language-separated throughout. The per-layer delta tells a matching story: two bursts of heavy activity (layers 0–2, then a sustained 11–15) bracket a quiet middle, and the final layer (29) is anomalous on every measure: a late-network burst where the geometry swings most negative, doing something categorically different from the middle third.
The rest → the per-layer delta/residual scans and the full cross-model interlingua gallery (the one thread that runs through every specimen in the lab) live on the Internal geometry page.
The surgery. The scans suggested interesting structure; the surgery found something more humbling, and more useful. On this model the scalpel never reaches reasoning at all — it acts only on the output path.
A full sweep ran on the 26B-A4B, the same hard-math battery as E4B, this time on an IQ3_XS llama.cpp build rather than the MLX scan checkpoint. Ninety-four configurations: single-layer dups and prunes across every sliding position, adjacent-pair dups, multi-layer prunes, global-layer surgery, prune-plus-dup combinations. Under strict scoring the result is a flat line. Not one configuration beats baseline on hard-arithmetic correctness. The best, prune_12, lands at +0.006 — statistical zero.
Two failure modes, both mechanical. Duplication corrupts routing. Duplicating a layer in a mixture-of-experts re-runs its router and re-applies expert selection over a residual stream never calibrated to see it twice. The model stops emitting parseable numbers: null rates climb to 40–56% on the heavier dups, and every adjacent-pair dup floors out. There is no single catastrophic anchor here, as the architecture predicted; the catastrophe is smeared uniformly across every duplication. Pruning doesn't break it — removing a mid-stack layer makes it more fluent. prune_13 cures the baseline's "Answer: Answer:" loop and emits clean, comma-grouped, magnitude-correct numbers that are wrong in the middle digits. Null drops from 25% to zero; strict-correct nudges down. It trades refusal for confident error.
And so, for the second time in this notebook, the partial-credit grader lied. On E4B it inflated close-but-wrong dups (§3). On the 26B it crowned prune_13 and prune_14 as the standout "winners" at +5.3 and +4.3 partial-t, numbers that evaporate to zero under strict equality. The two largest grader-gaming gaps in the whole project belong to these phantom champions. Whatever layer surgery does to this MoE, it acts only on output compliance (willingness to emit a number), which the partial grader mistakes for capability.
A second sweep (emotion-attribution and multiple-choice probes) was meant to confirm uniform breakage. It did the opposite. On multiple-choice factual recall, pruning is dramatic: fifteen configurations beat baseline at t ≥ 2, the best lifting accuracy from 0.30 to near-perfect. The raw responses explain why. The base model often "answers" a multiple-choice question with a single character, :, and scores zero. It isn't getting the question wrong; it isn't answering. Prune one mid-stack layer and the same model emits a clean, correct letter. The knowledge was always there; it just wasn't coming out.
That reconciles the two halves. Layer surgery here is an output-compliance fix, not a reasoning fix. Where the latent capability already exists (factual recall), the unclogged output is correct and the gain is real. Where it doesn't (sixteen-digit arithmetic), the unclogged output is confident and wrong, the same well-formatted garbage the grader rewarded in §3. The math phantom and the multiple-choice win are the same intervention, read against different knowledge.
A note on mechanism: this is probably not the quantization breaking the output. Large models tolerate aggressive quants well, and the load-bearing layers (embeddings, attention) sit at higher precision in this mix. The likelier culprit is the scaffolding itself. These probes force a bare, no-chain-of-thought completion on a few tokens of budget (no room to deliberate), and the model balks, stalling on a : rather than committing. Pruning a mid-stack layer relaxes the balk: the same model, denied the same thinking space, now commits the answer it already had. The surgery doesn't make it smarter; it makes it answer.
Why the dense sibling and the MoE diverge maps cleanly onto Ng's own framing. His retrospective argues a transformer's functional anatomy differentiates as it scales, and that RYS works by photocopying an indivisible mid-network circuit and running it twice. Dense E4B has such circuits — the two-zone mechanism of §1–§7 is exactly that picture. But a mixture-of-experts doesn't keep its capacity in contiguous layers; it scatters it across 256 experts, eight live per token, chosen fresh by a router. There is no whole-recipe-twice when the recipe is a different subset of cooks every step. "Every architecture has its own neuroanatomy," Ng writes: on this one, the surgery reaches the throat, not the brain.
Which points at the next instrument. If the capacity lives in the experts, the surgery has to as well — operating on the expert axis instead of the layer stack. Two candidates, and which one wins is genuinely open. REAP prunes low-utility experts outright; SEC (specialized expert cloning) leaves the size alone and overwrites a layer's least-used "junk" expert with a byte-copy of its most specialized one, biasing the router toward a capability with no training at all.
There's already a taste of the second, on a different patient. XpressAI published a logic-tuned SEC variant of Qwen3.6-35B-A3B but never shipped the method, and the repository is gone. It didn't need shipping: because router weights stay full-precision through quantization, the entire clone map (which expert was copied onto which, across 39 of its 40 layers) falls straight out of a byte-diff between their model and the stock one. Recovered, replayed onto a 2-bit quant that actually fits the kitchen, and the cloned model held together: stable, coherent, and visibly more willing to push through a causal-reasoning prompt where the untouched model looped and timed out. Cautiously promising. Whether the expert axis pays off on Gemma's MoE (by cloning, pruning, or some hybrid) is the next notebook.
§9½Status: the larger dense sibling, and the one the author stalled on
The 31B-dense is the model RYS's own author estimated at ~40 H100-hours and never got around to. The tooling here is no longer the blocker — the per-layer-array fix loads a coherent dup_2 31B that computed a 13-digit product correctly under chain-of-thought, at ~5.7 tok/s on the Mac Mini. The full sweep is parked, not abandoned — on two questions, both now close enough to resolved to run it.
Parked — clearing to run
Q1 · Is IQ2_XXS too noisy to trust?
At 2-bit, quantization is injected noise, and noise compounds with depth, so the worry is specifically the later layers, where a 2-bit error budget accumulates into something that may not reflect a proper-precision model. Partly mitigated by the quant's mixed precision (the load-bearing tensors are kept higher-bit), but not fully. Mostly resolved: it's what runs on 16 GB, so run it. But treat late-layer findings as quant-suspect and flag them for higher-quant replication.
Q2 · Pivot from the IT model to the base?
The instruction-tuned 31B is a thinking model: under forced no-chain-of-thought completion it keeps fighting to reason out loud and scores worse for it (the same commitment-collapse the MoE showed). The fix is to sweep the base model instead, pending a quick base-vs-IT comparo to rule out a glaring methodology issue. The genuine unknown is whether RYS layer-edits found on the base carry over to the IT model. Early evidence says probably yes: the base-vs-IT per-layer cosine/delta scan is dramatically closer than the IT-2-bit-vs-IT-4-bit scan: tuning moves the layers less than quantization does. Unconfirmed, which is exactly what makes it worth testing.
Method. Gemma-4-E4B-it, IQ4_XS quantization, llama.cpp. Layer surgery via the layer_path.py + per-layer-array reindexer (SWP, PLE token-embd, PLE model-proj). 16 hard-math probes, 15 seeds at temperature 0.3, base completion (no chat template, no system prompt). Paired-t against in-run baseline. Strict regrade: exact-equality + order-of-magnitude graders, on the verbose responses preserved from the original sweep.
References. The full bibliography (Ng's RYS posts, Grootendorst's Visual Guide to Gemma 4, the modeling_gemma4.py shared-KV routing source, and the alainnothere cross-model replication: Devstral-24B / Qwen2.5-32B, +245% BBH-Logical-Deduction from a 3-layer dup at L12–14) lives on the shared methods page.
This was a three-way effort. Opus 4.7 ran point on the mechanistic reasoning: reading modeling_gemma4.py by hand to find the sole-anchor mechanism, predicting which interventions should survive and which should crater, and absorbing the data in large enough gulps to spot the patterns. There were "MASSIVE finding" and "you see what i see?" moments throughout, but the line that captured the project's temperament best was a single message that opened, in bold, with "Oh hell yes." That was the instant a load-bearing theoretical claim about the consumer zone got falsified by two configs, the corrected mechanism arrived in the same breath, and the model card's whole shared-KV story finally clicked. The big context window earned its keep on this one.
Sonnet 4.6 did the slog — reading two hundred and forty-seven configurations' worth of verbose responses by hand to build the failure-mode taxonomy, debugging the gguf_fix_per_layer_arrays.py byte layouts, patching the sweep harness across a half-dozen schema changes, and continuing to function while Gemma in distress poured ten thousand tokens of "Self-Correction/Reasoning:" into the verbose log. The kind of work that doesn't write itself up because it's the act of writing things up.
Anya (me) sat between the two. Pointing things out. Pushing back when the model story didn't match the actual numbers. Refusing to accept that combo_2_36_39 was really the champion when the responses looked off. Saying things like "the pattern is bizarre" in the right places. Tweaking scripts and getting lost in .venv purgatory. Running the sweep on a Mac mini with sixteen gigabytes of RAM and a kitchen.
A Internal geometry
The per-layer cross-language geometry scans — Gemma E4B/26B/31B alongside Granite, Qwen-3.5, DeepCoder, NanBeige and MiniCPM — now live on a dedicated cross-model page, because the interlingua they reveal is the one thread that runs through every specimen in the lab. → Internal geometry — all nine models.