Local AI models, torn down, modified, and tested.

The model didn't produce an unambiguous success, but the instrumentation was built here, and one empirical regularity held up across every later specimen.

The one specimen where inference-time surgery measurably helped, albeit in a roundabout way.

This page showed me just how far my instrumentation had come, and it required more than just a few edits to address. Both the original and addendum update coexist to demonstrate how a rigorous approach matters. And, of course, the revised result isn't kind to the hypothesis of altering the granite model.

The original dissection and published page, and one of the larger models to receive an extensive sweep of its layers. This model also demonstrated the fragility of the evaluator and the necessity of interpreting the results correctly. The lenient grader handed back a tidy list of winners, but the strict regrade culled most of them. This model also served as a proof of concept as RYS layer edits don't play well with models structured like this. Unlike the dense isotropic models where each layer just interfaces with its neighbors, this model required some careful editing and remapping to work correctly, and certain layers remain off limits.

This is a report of a 176-config dup/prune sweep of a flagship 31B model squeezed into 16 GB at IQ2_M, and not one surgery won. This model stressed my machine and my patience, but given its size I had hoped to finally see a notable result. The original method suggests that a model with more layers develops a real language-independent section of layers that could be duplicated to improve reasoning without disturbing its processing. These findings suggest minimal to no improvement.

Every other experiment treats a model as a stack of layers, but there may be a different lever to try on mixture-of-experts models. This isn't a new idea either, and it has more institutional support in the form of Cerebras REAP, but I'm curious what happens when you take it further than they suggest.

A Hierarchical Recurrent Model sounds like the perfect experimentation platform for me. After all, my whole hypothesis is increasing reasoning depth through looping layers rather than just making a bigger model or training it. This is a novel class of transformer LLMs that's not ready for prime time yet, but that didn't stop me from exploring the exact promise of looped transformers - a built in system to multiply the number of passes the token takes through layers. The first overnight test delivered outstanding and exciting results, but as is a recurring theme here, upon further inspection and more rigorous testing, it turned out to be a fluke.

The RYS literature says smaller models don't develop a convergent reasoning zone in the middle layers. I scanned them anyway: the mid-stack zone shows up fine — what never shows up is any connection to RYS results, since the winning configs sat on layers the scans didn't flag. Still some interesting standouts, namely in the Gemma 4 family.

This brand-new dense Gemma model dropped right as I was refining the MLX hidden-state extraction script, and it seemed like a perfect opportunity to crack the model open before even a single fine-tune came out. The scan was interesting, but I took it further by investigating per-token hidden states - comparing two sentences that differ by barely a word but whose meanings diverge widely. This created a visual demonstration of where in the model a token takes on meaning, rather than being a number corresponding to a dictionary entry.

Anatomy of a sweep

Running the surgery isn't difficult once you establish the mechanics of it; extracting and evaluating the resulting model's response is where it turns into work. Three traps do most of the damage: a lenient grader inflates partial credit, the model can echo the answer straight back, and format-compliance passes for capability if you don't check. So before spending compute I profile how much headroom the baseline even leaves to measure, then calibrate the grading channel against it.

MLX is the fastest local runtime*

The asterisk is the whole point: 1B dense model, batched decode only, M4 Mini, custom generation loop, raw tok/s - every qualifier is on the label. The more interesting result, outside that narrow use case, is that MLX and llama.cpp are pretty evenly matched, so there isn't really a headline about which runtime is better.

duologue - one model, two agents

Duologue is more in the spirit of the model experimentation in the previous sections than a functional use for agents, but it did submerge me into the tricky world of context management and orchestration. It started with two models in RAM concurrently. I can't run two Qwen-27Bs at once, though, so the trick became feeding a single model a carefully crafted history that makes it believe it's talking to someone else. That resolved the mechanistic aspect. The fun part ended up being setting up various scenarios for it: first to see whether it recognizes the other agent as AI, and later whether the two agents could communicate under strict restrictions on their messages to each other.

mem0-processor

The project that kicked off everything else here. I wanted a persistent memory store kept current by LLMs, but the options were flawed: claude.ai's user memories are occasionally wrong, Claude Code has the model write its own unreliably, the two don't even talk to each other, and everything I found leaned on paid Claude to generate the memories. So I built a pipeline that uses free open-weight models to summarize my sessions and upload them to mem0, keeping close control over the transcript and prompt. Trusting those local summaries meant scoring the models to see which were any good, and that scoring rig quietly became a real evaluation harness: 119 models scored out of a 145-model registry, 62 clearing the quality bar.

Teaching an agent to read its own logs

What do you do when you have 1200 session summaries from hundreds of different local models sitting on your storage drive from testing the mem0 pipeline? I couldn't let that go to waste, and I'd been meaning to optimize my CLAUDE.md empirically from past experience. The summaries are meant to be small enough to parse easily, but cumulatively they add up to a corpus too large for any model to process at once. Through several layers of aggregation, I used Claude to distill them into meaningful observations that measurably improved my system prompt and memory system, to the point where starting a new session can feel frictionless.