Draft This specimen is a recurrent architecture I don't yet fully understand, and the headline finding here was overturned by my own verification pass. It's published as an honest work-in-progress — the negative result and the methodology lesson are the point. Read it as a notebook entry, not a paper.
Observation notebook · Anya · 2026 · the collapsed-champion branch
HRM: a false dawn.
SpecimenHRM-Text-1B · recurrent · MLX bf16 · 2×3 H·L cycles = 8 passes / token · 128 unrolled layers, weight-shared · 35 configs · 5 waves · then the verification that killed it
StatusA recurrent reasoner, and a result that didn't survive — "champion" 5/6 (greedy, n=1) → 0/3 sampled, verified
Sourcesapientinc/HRM-Text-1B · Aryagm/HRM-mlx port

A Hierarchical Reasoning Model — a recurrent architecture whose whole thesis is reasoning-depth through loops, not parameters. The dissection produced a clean mechanistic picture, one gorgeous finding ("duplicate the renderer module and multi-hop reasoning improves"), and then — when I ran the rigorous follow-up the finding deserved — the headline collapsed. Sampling killed it. What survived is more useful than what died.

The patient

HRM-Text-1B is a language-scale adaptation of Sapient's Hierarchical Reasoning Model — two recurrent transformer stacks, a low-level L module and a high-level H module, iterated together. The original was ~27M parameters and reasoned on ARC and Sudoku purely by recurrent computation at fixed parameter count. Run here through Aryagm's HRM-mlx Apple-Silicon port — which is not mlx_lm-compatible; the recurrence needs bespoke per-pass KV caches.

The probe arsenal: a cross-language interlingua scan (geometry), a per-cycle logit lens (mechanism), a coherence detector (degeneracy floor), and a small reasoning battery (capability). Triangulating them is the whole method — and is also exactly what went wrong.

§1The slow/fast split is real, and legible

The forward loop, almost literally: the fast L module relaxes many times toward a target held fixed by the slow H state, then H takes a single step, moving the target, and repeats. It is nested fixed-point iteration: mutual refinement of one evolving state, not a one-way pipeline.

A per-cycle logit lens (decoding each module's state through the output head) makes the division of labour visible in plain token space. H (meaning) decodes to content: the problem's actual numbers, reasoning words like "Since," "We," "By." L (surface) decodes to subword fragments that aren't clean output tokens. The roles aren't a training story you have to take on faith; they're right there in the decode.

Depth geometry — 128 unrolled layers Adjacent-layer cosine (left axis, solid) and delta-norm (right axis, dashed) across all 128 unrolled-layer transitions of the baseline HRM forward pass — two cycles, each cycle three L-submodules then H, 16 layers per module. A smooth residual highway sits at cosine 0.85-0.99 through most of the body, punctured by eight sharp dips exactly at the module-boundary transitions (marked with vertical rules), where cosine falls as low as 0.55 and delta-norm spikes as high as 399. A secondary panel below isolates the last pass's H module (16 states) against its last L submodule (16 states): H's per-layer cosine spans a wider range than L's, consistent with H being noisier per layer than L late in the pass. Depth geometry — 128 unrolled layers, one prompt Adjacent-layer cosine (left axis, solid) and ‖Δ‖ (right axis, dashed) baseline config, 2 cycles × (L0,L1,L2,H) × 16 layers cycle 0→1 (c0.H→c1.L0) 0.5 0.6 0.7 0.8 0.9 1.0 0 100 200 300 400 1 64/65 128 transition # (of 128) adjacent cos (left) ‖Δ‖ (right, dashed) module boundary adjacent cos ↑ ‖Δ‖ ↑ Last-pass per-module depth (cycle 1) — H (16 states) vs L2 (16 states, last L submodule) 0.80 0.90 1.00 l0→1 layer transition (1 of 15, last pass) l14→15 H, last pass (cycle 1) L2, last pass (cycle 1) Final adjacent-cos in this run: H 0.872, L2 0.898 — both plotted verbatim above, neither hits the caption's 0.71.
Depth geometry — 128 unrolled layers Adjacent-layer cosine (left axis, solid) and delta-norm (right axis, dashed) across all 128 unrolled-layer transitions of the baseline HRM forward pass — two cycles, each cycle three L-submodules then H, 16 layers per module. A smooth residual highway sits at cosine 0.85-0.99 through most of the body, punctured by eight sharp dips exactly at the module-boundary transitions (marked with vertical rules), where cosine falls as low as 0.55 and delta-norm spikes as high as 399. A secondary panel below isolates the last pass's H module (16 states) against its last L submodule (16 states): H's per-layer cosine spans a wider range than L's, consistent with H being noisier per layer than L late in the pass. Depth geometry — 128 unrolled layers Adjacent cos (left axis, solid) · ‖Δ‖ (right axis, dashed) baseline config, 2 cycles × (L0,L1,L2,H) × 16 layers cycle 0→1 0.5 0.6 0.7 0.8 0.9 1.0 0 200 400 1 64/65 128 transition # of 128 (embed→c0.L0.l0 … c1.H.l14→l15) adjacent cos (left) ‖Δ‖ (right, dashed) Last pass (cycle 1): H vs L2 per-layer 0.80 0.90 1.00 l0→1 layer transition (last pass) l14→15 H, last pass L2, last pass Final adjacent-cos this run: H 0.872, L2 0.898 — plotted verbatim, neither hits the 0.71 caption.
Fig 1 — depth geometry. Adjacent-layer cosine and delta-norm down the unrolled stack. A smooth residual-stream highway (cos 0.96–0.99) through the body, then a violent rotation into output-token geometry at the final layer (cos→0.71, ‖Δ‖ spike). The L (fast) module is noisier per layer than H (slow). Same signature as Granite and Gemma — a textbook transformer wearing a recurrent costume.

§2An interlingua — with a warning label

Using a cross-language centered-cosine probe (parallel EN/ZH/FR sentences), the model develops a partial language-decoupled "interlingua" — but only for structured content, and only in the H module.

Math, English↔Chinese: cosine jumps to +0.32 the instant the state enters the H (slow) module, cycle 0, and recurs across cycles. Prose (fact, poem) EN↔ZH never crosses (~0). Only structured, numeric content becomes language-decoupled, and it happens in the meaning module, exactly where the lens says meaning lives.

§3The loop-scaling negative (and a self-correction)

HRM's whole premise is buying reasoning with more loops, so I pushed H_cycles from 2 to 8. It does not help: the math interlingua peaks at the trained cycles and decays to noise by H2+, and the H-state does not converge to a fixed point: it oscillates.

I first logged this as "low-rank collapse." An effective-rank probe falsified my own claim: the participation ratio rises across cycles (10→22), it doesn't fall. The real mechanism, from the lens: over iteration, the meaning channel converges onto end-of-sequence and formatting tokens (<|box_end|>, \n). The model reads excess compute as "you're done, terminate." Content evaporates into termination, at full rank. The checkpoint is saturated at its trained depth; you cannot buy reasoning with free test-time loops. (First lesson that a probe can mislead: a rank probe corrected a wrong reading of a cosine probe. It would not be the last.)

§4The exciting finding — a sweep, and a "champion"

A unified harness, ~35 configs across five waves: layer duplication, per-cycle L-schedules, residual damping, H/L ratios. Readout per config: geometry + interlingua + logit-lens + a 6-item reasoning battery — scored greedily, n=1. A clean story emerged.

configbatterysolved the multi-hop age-sort?
baseline3 / 6✗
rys_l  (duplicate early L layers)5 / 6✓  — best of all configs
combo_rys_anchor5 / 6✓
ratio_hi_l  (more L cycles)4 / 6✓
ext_h2_l6  (2× L)4 / 6✓
every extended-H config0–1 / 6✗

The pattern was gorgeous and theoretically motivated: more settling in the fast/renderer module helped reasoning; more drift-inducing H hurt. A narrow sweet spot, early-L duplication. And the interlingua geometry seemed to corroborate it. It even survived a first robustness pass: the supposed null control (duplicating L) was the single best config. (In hindsight, that the control won should have been the tell.)

The config sweep — interlingua strength vs battery score Scatter of 29 configs that have both an interlingua measurement and a battery score on disk (of the page's stated 35; six configs — anchor_first, combo, damp_L, damp_native, hh_damp — have no battery file at all, so a 35-config sweep on this axis cannot be drawn). x is each config's peak per-cycle math-interlingua cosine; y is the mean pass-rate across that config's battery items. Points are drawn as circles for configs scored greedily at n=1 and squares for configs whose only on-disk battery file is the later n=3-sampled verification run — a scoring-protocol mix, not a single consistent battery. Under this reading only one config, ext_h3, shows a perfect age-sort score; the three configs the page text calls the champions (rys_l, ratio_hi_l, combo_rys_anchor) are all sampled-run points and do not solve age-sort in the data. The config sweep — interlingua strength vs battery score 29 of the stated 35 configs have both files on disk · x = max per-cycle math-interlingua cosine · y = mean battery pass-rate 0.0 0.1 0.2 0.3 0.4 0.5 interlingua strength (max per-cycle math cosine) → 0.0 0.2 0.4 0.6 0.8 battery score (mean pass-rate) ↑ baseline rys_l ratio_hi_l combo_rys_anchor ext_h3 ✓ solved ext_h4_dampH l_h2_l16 greedy, n=1 (original sweep score) sampled, n=3 (§5 verification-run file) age_sort solved (rate/pass = 1.0) Solved configs scatter across the x-range — geometry/reasoning decoupling holds. But the "champion" cluster (rys_l, ratio_hi_l, combo_rys_anchor) only has verification-run battery files on disk, and none solve age-sort there. Item sets differ too: 6 items (greedy) vs 10 (sampled, +runners/seating/meetday/bus) — y averages what each file has.
The config sweep — interlingua strength vs battery score Scatter of 29 configs that have both an interlingua measurement and a battery score on disk (of the page's stated 35; six configs — anchor_first, combo, damp_L, damp_native, hh_damp — have no battery file at all, so a 35-config sweep on this axis cannot be drawn). x is each config's peak per-cycle math-interlingua cosine; y is the mean pass-rate across that config's battery items. Points are drawn as circles for configs scored greedily at n=1 and squares for configs whose only on-disk battery file is the later n=3-sampled verification run — a scoring-protocol mix, not a single consistent battery. Under this reading only one config, ext_h3, shows a perfect age-sort score; the three configs the page text calls the champions (rys_l, ratio_hi_l, combo_rys_anchor) are all sampled-run points and do not solve age-sort in the data. The config sweep 29 of 35 stated configs have both files · x = peak interlingua cosine · y = mean battery pass-rate 0.0 0.2 0.4 interlingua strength → 0.0 0.4 0.8 battery score ↑ baseline rys_l ratio_hi_l combo_rys_anchor ext_h3 ✓ greedy, n=1 (original sweep) sampled, n=3 (§5 verification file) age_sort solved (rate/pass = 1.0) Solved configs scatter across x — decoupling holds. But rys_l, ratio_hi_l, combo_rys_anchor only have verification-run files on disk, and none solve age-sort there. Battery item sets differ: 6-item configs (greedy) vs 10-item (sampled, +runners/ seating/meetday/bus) — y is the mean over whatever items that config's file has.
Fig 2 — the 35-config sweep (greedy, n=1). Each point a config: interlingua strength (x) vs battery score (y); solved-the-age-sort configs marked. Two things are visible at once — and only one of them survives §6. The solved configs scatter across all interlingua values (the claim that geometry is decoupled from reasoning — this holds); and they cluster in the more-L configs (rys_l, ratio_hi_l, combo_rys_anchor — the "champion" story that does not). This is the figure that looked like a result.

§5The turn: verification deflates it

The whole thing rested on greedy, n=1 scoring. The rigorous follow-up — sampling at temperature 0.7, N=3 per item — broke it immediately.

age-sort, sampled (N=3)scorevs greedy
baseline2 / 3solves it ~70% on its own
rys_l  (the "champion")0 / 3worse than baseline ←!!
combo_rys_anchor1 / 3within noise
ratio_hi_l2 / 3within noise
ext_h2_l62 / 3within noise

The champion scored 0/3, worse than baseline. No config beat baseline; everything non-extended clustered within N=3 noise. The base model is simply non-deterministic and solves these puzzles ~70% of the time on its own: the greedy sweep had been measuring which single draw got lucky, not capability.

And the interlingua metric that "corroborated" the story? Decoupled. Across all configs the highest-interlingua runs were among the worst behaviorally; a config with near-zero interlingua reasoned fine. The pretty geometry measures a representational property, not capability: the §2 warning label, cashed in. (Second, decisive lesson that probes mislead: this one had fooled me for a full overnight.)

§6What actually held — and the mechanism

Reading the real generations and the per-cycle lens — not just the scores — recovered a satisfying result: the modules' roles, confirmed by how they fail.

ext_h4 — more Hfailure mode: drift off-task
H is the integrator — over-run it and it loses the thread
z_h, cycles 0–1: [names, task content] ← identical to baseline z_h, cycles 2–3: ['1','To','From', …] ← untrained cycles, names vanish output: number / symbol salad
The first two H-cycles match baseline; the untrained cycles drift the meaning state off the task tokens, and the output degenerates. Extra H-updates bury the input under accumulated abstraction.
rys_l — duplicate Lfailure mode: surface fragmentation
L is the renderer — over-drive it and the surface destabilises
z_h fragments: 'C','Be','A' instead of 'Eve','Cara','Ben' generations: ordering CORRECT, but Cara→"Costa"→"Rice", answers truncate
The reasoning ordering is usually right; the names get corrupted and answers truncate. So rys_l's "0/3" is largely a rendering defect an exact-match scorer punishes as a reasoning failure — a subtlety the score alone hid.

The modules behave exactly as their roles predict: the surgery perturbs the surface (L) or the state stability (H), and neither reliably improves reasoning. The robust conclusions: you can reliably make this model worse (extend H → garbage); you cannot reliably make it better by inference-time surgery; one fencepost trap (clock_strike) was solved by nothing, ever, a hard capability ceiling. The reasoning capability is near-fixed and saturated; the real lever is retraining, not surgery.

§7What each instrument is actually good for

probemeasuresverdict
interlingua (cosine)representational geometrydecoupled from capability — not a capability proxy
coherence (distinct-trigram + boxed-rate)degeneracy / non-commitmenta crater floor detector (Pearson +0.44); can't separate good from better
logit lens (per-cycle)mechanismthe MVP — pinpointed drift vs fragmentation; twice corrected wrong readings
behavioral batteryreasoningthe only capability signal — but worthless at greedy n=1; needs sampling + headroom + N≥~8

§8The methodology lesson

A high-variance model + greedy single-shot evaluation + a corroborating-but-decoupled geometric metric will manufacture a clean, theoretically-satisfying, completely false finding — and it will survive a casual robustness check. Sampling and champion-verification are not optional polish; they are the difference between a result and a hallucination. A metric that agrees with your story is not evidence until you've shown it tracks the outcome you care about. HRM-Text-1B · the real takeaway · 2026

Caveats

The verification pass that killed the headline finding is itself thin: N=3, on a battery calibrated for within-model deltas, not a cross-model claim. Read what follows as the honest state of an already-walked-back result, not fresh doubt on a stable one.

Build notes & method

Model: Aryagm/HRM-Text-1B-MLX (bf16 probing / 4bit serving), via HRM-mlx. Always --static-cache; bf16, never fp16. Scripts: extract_states.py · interlingua_scan.py · logit_lens.py · experiments.py (sampling) · regrade.py (strict boxed) · probe_coherence.py. Full-generation JSONs in figs/. Sibling specimens: Granite · Gemma 4. Shared methods & references.