One probe, run on every specimen: take parallel sentences in different languages that say the same thing, and measure the cosine similarity of their hidden states at each layer. Where a model holds a shared concept space, same-content-different-language pairs converge in the middle of the stack — an interlingua — while the input and output ends stay in language-specific token geometry. This scan exists because of a hypothesis it was supposed to serve: if models develop “reasoning layers,” the scan should find them, and RYS duplication should pay off where the plateau sits. On these small models it doesn’t cash out — most of the winning configurations were active on layers the scans never flagged, and the data collected here doesn’t suggest any correlation between the interlingua zone and where duplication helps. What survives is the map itself: the mid-stack plateau lands in the same region the lesion maps flag as where meaning lives, and it shows up quant-invariant on every model below.
§How to read it
Each line tracks how similar the model’s internal snapshots of two phrases are, layer by layer (higher means the model treats them as more alike at that depth). At the ends, the ranking is boring: cross-language pairs sit at the bottom because they are literally different (different scripts, different token counts), first as raw input, then again as the model rebuilds the output language. The middle is where something interesting can happen: pairs that look nothing alike but mean the same thing climb toward each other, which suggests the model is translating both into an internal working representation of the meaning, plausibly not any human language at all. The tell isn’t convergence by itself (deep in a network everything drifts somewhat closer; the scans are centered to cancel exactly that); it’s same-meaning pairs rising above the different-meaning baseline.
The delta charts are the second lens: how much each layer actually moves the hidden state (‖h_L − h_{L−1}‖), a rough measure of how much work it does. The ends always spike (embedding handoff on the way in, generation mode on the way out); what matters is the middle, where low-delta layers barely touch the stream. Those are the natural surgery candidates, since duplicating or removing a layer that changes little should hurt little — so where the cosine scan maps what the mid-stack holds, the delta scan maps how hard each layer works on it. Read it as a candidate picker, not a validated predictor: it tells you where to try first, not where you’ll win.
The line that matters is same-content, cross-language (e.g. EN ↔ ZH fact). When it rises above the same-language-different-content baselines in the middle layers, the model is representing meaning independently of language there. Where it peaks, and how sharply it collapses back to token geometry at the output, is the structural signature each architecture leaves.
Three things recur across the nine models: the interlingua is mid-stack, not distributed; the ends are token I/O; and the whole picture is quant-invariant, since 2-bit and 8-bit scans of the same model superimpose. The geometry is a property of training, not precision. The one thing that does not recur: any relationship between this map and RYS results — the duplication configs that won their sweeps mostly sat on layers no scan pointed at.
§The gallery
Expand any model. Cosine-similarity scans are centered (per-layer mean removed) and read on the last token unless noted; delta charts show per-layer hidden-state change.
Gemma 4 E4B · 42 layers 4B MoE — anchor zone at L22–L23
Fig G1a. Cosine similarity vs baseline by layer, last-token hidden states. Note the sharp polarity shift at layer 24 (post-anchor zone): cross-lang same-content pairs that were weakly positive invert to weakly negative, while the paraphrase signal halves. The anchor zone (L22–L23) is the discontinuity.
Fig G1b. Same scan using mean-pooled token representations. The cliff is visible but softened; the cross-lang signals shift from ~0.15 positive (layers 1–22) to near-zero or slightly negative in the consumer zone, rather than inverting sharply. Paraphrase similarity drops from ~0.55 to ~0.44.
Fig G1c. Per-layer hidden-state delta magnitude (‖h_L − h_{L−1}‖, token-averaged across cosine-scan inputs). Layers 0–1 are the embedding handoff spike; layer 23 shows the anchor-zone encoding burst. Consumer zone (L24–L41) settles to low, stable delta.
Fig G1d. Per-layer Δ/‖h‖ ratio split by input confidence group (high vs low). Both groups track together until the anchor zone, where they briefly diverge before reconverging in the consumer zone. The convergence suggests confidence is re-encoded rather than passed through.
Gemma 4 26B-A4B · 30 layers native mixture-of-experts, ~4B active/token — scanned on the heretic-msq base checkpoint, pre-surgery
Fig G2a. Cosine similarity by layer for the 26B-A4B (centered last-token states). Noisier than E4B's — cross-language same-content pairs never cleanly separate from the diff-content baselines. Consistent with the surgery finding: this model's capacity lives in expert routing, not in layer-addressable structure, so there's little for a layer-wise probe (or scalpel) to grab.
Fig G2b. Per-layer delta magnitude for the 26B-A4B. Flatter than E4B through the mid-stack with a late burst — and no anchor-zone spike, because this architecture has no shared-KV anchor (that was E4B-specific, not a general Gemma trait).
Gemma 4 31B · 60 layers Gemma 4 31B · 60 layers · dense (3-bit MLX scan) — the original RYS thread's stalled target
Fig G6a. Gemma 4 31B cosine geometry (centered, 60 layers, full probe set). The curves stay bunched near the top through most of the stack and only fan out past L50, the math pair holding cross-language similarity longest — a denser, cleaner version of the E4B story.
Fig G6b. Per-layer delta for Gemma 4 31B. Rhymes with E4B: very-low-delta layers early (L2, L4) and a low-delta mid pair (L13–14) on a high-residual plateau (L8–22), bracketed by high-delta entry layers and a sharp generation-mode spike at the final layer. No shared-KV anchor → no single catastrophic point; a wide band of safe-to-duplicate mid-stack layers, low-delta candidates first. Surgery pending.
NanBeige-4.1 3B · 32 layers ~3B parameters — strong multilingual signal
Fig G3. NanBeige cosine geometry (6 probe pairs; no FR/math/paraphrase in this scan). The extreme is concentrated at L3: cross-language same-content poem similarity peaks at +0.92 while same-language diff-content (ZH) plunges to −0.99 — the sharpest single-layer multilingual clustering of any model here. It decays through the early layers rather than persisting, suggesting dense, shallow concept encoding.
MiniCPM v4.6 · 24 layers MiniCPM v4.6 · 24 layers · 1B (Qwen3.5-0.8B + multimodal base)
Fig G4. MiniCPM v4.6 cosine geometry. Cross-language same-content pairs reach moderate positive similarity (~0.54 peak, L5–6); diff-content cross-language pairs stay strongly negative (−0.75 to −0.90), clean separation. Note the embedding-layer (L−1) anomaly: same-content pairs are most dissimilar at the embedding — a tokenizer-level language divergence the network resolves upward.
MiniCPM v5 · 24 layers MiniCPM v5 · 24 layers · 1B (newest generation)
Fig G5. MiniCPM v5 cosine geometry. A concentrated spike at L4 (same-content poem → +0.90; poem-vs-fact → −0.97) — the NanBeige pattern, sharper: a single-layer burst of multilingual discrimination, then decay. The same-content fact signal is weaker than v4.6's, suggesting whatever sharpened poem alignment traded off fact alignment.
Granite-4.1-3B · 40 layersdense · scanned at 2/4/8-bit
Fig G7a. Cross-language same-content cosine climbs to a broad mid-stack plateau (L15–30), the interlingua / concept space, while the two ends sit in token-I/O geometry (deeply negative at the embedding side, re-separating at the output). The lesion map from the Granite sweep is this same map: noise is most lethal exactly where meaning lives. Quant-invariant: the 2-bit and 8-bit scans are superimposable on this (4-bit shown); the geometry is a property of training, not precision.
Fig G7b. Per-layer hidden-state delta. The U-shape (large at the input/output ends, a flat ≈0.22 plateau through the middle) is the residual multiplier made visible, and the inverse of the noise-vulnerability curve.
Qwen-3.5-4B · 32 layersdense · scanned at 2/4-bit
Fig G8. Qwen-3.5-4B at 2-bit shows the same interlingua-in-the-middle signature (cross-lang same-content rising to a mid-stack peak, ends in token geometry) and it is unchanged at 4-bit (2-bit shown). Independent confirmation, on a different architecture and a harsher quant, that the concept-space geometry survives aggressive quantization.
DeepCoder · 48 layersdense reasoning model · scanned at 2/3-bit
Fig G9. DeepCoder (48 layers) carries the same mid-stack interlingua plateau across a much deeper stack, with the concept space spread over a longer middle band before the final-layer flip to output geometry. The signature is architecture-general; the depth over which it spreads scales with the model.
Scan tools + data: rys-tools/scans/ (mlx hidden-state extraction, centered cosine). Specimens: Gemma 4 · Granite · HRM. Shared methods & references.