Every other specimen in this lab treats a layer as the thing you operate on. Point the same machinery at a Mixture-of-Experts model and it stops working — and the way it stops working is informative enough to be its own result. This is the page where the line forks.
The working hypothesis coming in: a layer is the universal unit of surgery, duplicate, prune, or damp one and capability moves, on any architecture. It held across every dense specimen, Granite, MiniCPM5, where a layer genuinely is a unit of computation you can amplify. On an MoE it doesn't. Duplicating a block re-runs the router and double-applies expert selection on an uncalibrated residual: the router sees an input distribution it was never trained on, null rates explode, strict scores floor. It's not "no effect," it's "actively breaks the routing." Hypothesis retired for MoE: the unit of capacity here is the expert, not the layer, so the line forks to the expert axis, and the rest of this page is that work.
§1The wall
Whole-layer surgery has no purchase on a Mixture-of-Experts model, for two distinct reasons. Duplication corrupts routing: a duplicated MoE block is mechanically destructive. And pruning is the wrong axis, not harmful: pruning an MoE layer is an output-compliance fix, unclogging a stalled output valve so the model emits a clean letter or number, not a reasoning fix.
Whether the unclogged output is correct depends entirely on whether the latent capability was already there. On factual multiple-choice (latent), unclogging surfaces a real answer the model knew (a genuine, large gain). On 16-digit arithmetic (not latent), unclogging just produces confident, well-formatted garbage: the strict phantom. Same mechanism, opposite verdict, decided by latent-knowledge presence — the exact compliance-not-capability pattern, here as the whole story rather than a footnote.
The reason is structural: on an MoE the unit of capacity is the expert, distributed across hundreds of them (~8 active per token), not the layer. To touch what the model actually knows, you have to operate on experts.
↳ a layer is a computation you can repeat; an expert bank is a routed lookup you can't repeat without breaking the route
§2The expert-surgery toolkit
Three operations on the expert axis, three different jobs. REAP is a functional-importance cut: it measures each expert's data-conditioned contribution (gate weight × output, aggregated over a calibration set) and culls the low-saliency ones. That's a "what does the model actually use" cut, the opposite of the geometric cosine/delta scans the dense work uses.
| Technique | What it does | Use case |
|---|---|---|
| REAP · prune | delete low-utility experts (router-weighted activation pruning) | shrink total params |
| ExpertClone · merge | blend similar experts | densify, preserve knowledge |
| SEC · clone | overwrite junk slots with copies of specialist experts | bias routing toward specialists, no retraining |
REAP's published headline (Cerebras, arXiv 2510.13999) is 50% expert compression, near-lossless on code for very large MoEs (Qwen3-Coder-480B, Kimi-K2), with pruning beating merging on generative tasks because merging loses routing control. The lab's own datapoints below deliberately go deeper than that documented window, into the degraded-but-usable regime. The two concrete results that follow are an SEC replay and a REAP teardown; the lab's own runs of both are still ahead.
§3You can steal a clone map by diffing routers
Someone had surgically re-biased an MoE's routing toward its specialist experts,
shipped the result, and withheld the how. xpressai published a modified MoE (the live card is
Qwen3.6-35B-A3B-ExpertClone-Logic), but the implementation repo is not
publicly accessible (and not on the Wayback Machine as of 2026-06), so the profiling
methodology is permanently opaque. That opacity is what makes the router-diff method necessary: without it, the surgery couldn't be recovered at all.
You don't need their code to replay the surgery, because of one architectural fact: the
router tensor (ffn_gate_inp.weight) is F32 in every quant of
the same base. In the cloned file, the victim router columns were overwritten with the donor's — so
they now match the donor. Therefore a router column that matches base is a donor (untouched); a
column that differs from base is a victim (overwritten). Donor→victim direction falls out
definitively, no guessing.
Run that diff and you recover the clone map — 39 of 40 layers cleanly (layer 39 was genuinely untouched), with victim indices systematically small (0–17, the low-firing experts) and donors spread (1–229), which independently confirms a "lowest-total-firing expert as victim" selection rule. Then replay it: 39 × 4 = 156 mmap slab copies onto a runnable pre-quantized IQ2_M of the same base, near-instant, quant-agnostic. You've reproduced a gatekept model's surgery on hardware you own, without their code.
↳ victims hug the low-index slots across 39 of 40 layers; layer 39 is entirely donor
A strong clone doesn't visibly stumble on eyeball prompts — so the SEC delta is real-but-sub-perceptual, exactly where ad-hoc evaluation fails and the calibrated seed-paired battery earns its keep. The router-diff replay · §3 · capability eval still preliminary
§4Deep expert excision is usable below parity
You can't A/B a 100B→30B retention on a 16 GB Mac (you can't run the 100B base), so the locally-obtainable verdict is usability, not parity. There's one clean, unconfounded datapoint, read straight off a published model's own card.
huihui-ai/Huihui4-8B-A4B is a REAP of gemma-4-26B-A4B-it: experts
128→32 (75% culled), 8 active/token, all 30 layers
intact, a pure count-prune. The param math checks out exactly: base routed-expert 22.84B + fixed 2.40B
= 25.23B; cull to 32 → routed 5.71B + unchanged 2.40B = 8.10B, dead-on the "8B" label. The 26→8B ratio is only 3.1× despite a 4×
expert cut because fixed overhead (attention, dense FFN, embeddings, router, norms) doesn't scale with
expert count.
The unconfounded number, from their own perplexity eval: straight 128→32 prune, 0 SFT steps, PPL 1.596 → 2.483 (+55%). Degraded, but not broken: 75% of the experts are gone and it's still a functioning model. The baseline that makes that interesting isn't the unpruned model (you can't run that on 16 GB), it's non-existence. The real comparison is "a degraded version you can actually run" vs. "nothing at all," which is why a +55% PPL hit reads as intriguing rather than disqualifying: this is an accessibility result, not a capability claim.
↳ the only measured point sits past the estimated cliff, yet well above the broken floor
Grader-skepticism note: their post-SFT PPL drops below the 128-expert base (1.004 < 1.596), which tempts a read where 32 experts beat 128. That's false: the SFT and the PPL eval are on the same 500 samples, eval-on-train-distribution leakage (memorising a narrow thinking-format), not genuine superiority. The defensible claim is the 0-step +55% number; the sub-base number is an eval-leakage artifact and shouldn't be cited.
§5Why this is the live direction for consumer ML
A 250B dense model is dead in the water locally (250 GB working set). A 250B MoE with mmap + expert offload has a working set of ~10 GB, because only the active experts are hot: tractable at ~1–3 tok/s for cold-NVMe pages. MoE + REAP + local is the one regime where a consumer machine gets to touch frontier-scale models at all.
The standing hypothesis (speculative: plausible mechanism, untested in this lab): active params, not total, may be the intelligence proxy, up to the trained router-k. Pruning total to 30–40B while keeping ~6–10B active per token might gain per-token quality: remaining experts get hit more often, the router is less stretched. The supporting datapoint, a 35→14B-A3B reaped model "nearly keeping up" with full siblings, is an eyeball read, not a paired eval — so this stays a hypothesis, not a finding. The tax: a reaped older base loses to a newer dense one (generational gains > REAP preservation), so the workflow only pays on current-SOTA bases and has to be re-run per base update.
The engine direction matches the rest of the lab's evolution: the refined surgery line is
live-MLX-module patching, not GGUF slab-rewriting — which is also why ragged
per-layer expert counts (the elegant "let a global saliency threshold decide each layer's count" version
of REAP) are a dead end in stock GGUF (a single global n_expert scalar) and a natural fit
in MLX.
Caveats
The router-diff method is solid; what it hasn't produced yet is a quantified capability result. SEC extraction and the lab's own REAP run are still open: read this page as a working log of a pivot in progress, not a finished result.
- SEC capability is preliminary — extracted + replayed + vibe-tested, no fixed-seed paired battery run, no probe battery or standardized benchmark run yet. I have a premade SEC-edited model but not the SEC tooling itself, so reproducing the method from scratch is still needed; so far the only test has been extracting the merged experts' routing and running it ad hoc. The router-diff method is solid; the quantified recovery is the next step.
- The lab's own REAP run has not happened yet. The point of REAP is exactly that no training is needed — scan activations, prune the unfired experts, done. Huihui's authors considered a retrain because their prune was deep; the bet here is that a bare reaped instruct model stays coherent without one, which is what would let a model too big for this Mac at full size run locally in a degraded but still-capable pruned form.
- Huihui (the 26B→8B-A4B reap) isn't the model I'll actually test on. It's just the one I happened to find with a deep prune already applied. I've also run a barely-perturbed-feeling Qwen3.5-A3B pruned from 35B down to 14B. The plan: validate REAP on models small enough to run at full size first, then extend it to 60B+ models that only become usable locally once pruned.
- All numbers are point-in-time from memory. Fig 2's exact per-layer victim columns are illustrative; the clustering structure is the claim, not the specific indices.
The MoE branch of the vivisection lab — where the dense-model machinery (Granite, MiniCPM5) hits its limit and the unit of surgery changes from the layer to the expert. Router-diff runs against full-precision ffn_gate_inp.weight tensors in pre-quantized GGUFs; teardown numbers read off published model cards. Built and run locally on a 16 GB Mac Mini. The inferential harness that makes the sub-perceptual deltas measurable is the methods writeup.