I set out to build agent memory and built a model-evaluation harness instead. The memory is the use-case it was born from; the eval layer is the part that's reusable — and it surfaced a finding the pipeline was never built to look for.
§1One inversion is the whole architecture
Ask an agent to write its own memories during a conversation, and the memory-writing competes with the user's task for the model's attention. On every turn the model splits effort between answering you and persisting state, and persistence is the lower-urgency job, so in practice it gets a perfunctory sentence or gets skipped. So invert it: don't instrument the live session, mine the logs offline.
Claude Code already writes complete JSONL transcripts of every session it runs. After the fact, a local model batch-summarizes those transcripts, the summaries are scored against ground truth, and the survivors are uploaded as searchable memory. ETL, not live-write. That single inversion is the entire architecture; everything else is making it reliable.
The appeal: the source of truth is data you already have and never had to contaminate the live interaction to collect. The cost: you've now made writing good memory into a summarization-quality problem, which means you have to know which local model actually summarizes well, and that turned out to be a real question with no published answer. That cost is the pivot. Chasing "which model summarizes well" turned mem0-processor into a model-evaluation system that happens to emit memories as its output.
§2The core system: a summarization eval harness
Picking which 1–9B local model writes your memories is a model-selection problem, so the pipeline grew a benchmark, and the benchmark grew teeth. It carries 119 models scored (62 in the quality tier, 31 with enough shared sessions to rank head-to-head), evaluated on the exact task they'd be used for: not MMLU, not vibes, but "did the summary capture the facts this session actually established." Three disciplines do the work:
- Ground truth is hand-written fact sets, per reference session. A summary is scored on fact recall against a human-authored list of what the session genuinely contained — so the metric rewards capturing the thing, not producing fluent prose about it. The recurring failure it catches: the model that writes a beautiful, confident summary of facts that were never in the transcript.
- Ranking is by z-score across sessions, not raw averages. Session mixes differ wildly in how much there is to capture, so a raw mean over a model's sessions isn't comparable to another model's raw mean over a different set. Z-normalizing per session first makes the comparison legitimate. (The same "raw averages aren't comparable across different mixes" correction the RYS work makes with channel-balanced aggregates — identical bias, identical fix.)
- Sampler/config provenance is tracked. "Model X is good at this" without sampler provenance
doesn't replicate — the same claim at a different temperature can flip. The registry
(
config/models-registry.json, ~145 entries) carries the launch flags, chat-template fixes, and per-architecture reasoning-budget defaults that were each learned the hard way, so a result is reproducible from the registry key alone. The registry may outlive the pipeline.
§3What "capturing the facts" actually means
The scored target is not the shape of the session or its headline events. Those are close to free: Claude Code's own Haiku compaction pass already emits a high-quality summary of what happened. What this benchmark scores is the layer above that, decision attribution, meaning who made each call, who caught what, and whether a decision was contested or silently assumed. That's the layer later sessions actually need (the why outlives the what), and the layer models fall down on.
Session 2503c51c is the canonical hard case: three distractors wrapped around one real
bug. Mid-session, Claude proposed new Date() for a summary's timestamp header; the user
read the code and caught that this stamps processing time rather than session time,
and redirected the fix to pull startedAt from the transcript JSONL. The bug never threw —
it would have silently corrupted the timestamp on every future summary, undetectable until someone ran
temporal analysis. The two nearby exchanges are traps: a double-timestamp worry and a timezone
question were both cases where the user misunderstood and Claude was correct. A model that
pattern-matches "user raised a concern → there was a bug" inverts all three.
A summary scores well here only if it gets the attribution right, not the content: Claude was wrong, the user caught it by reviewing code, and the failure mode was silent corruption. That last clause is second-order (it requires reasoning about a downstream failure that never visibly happened), and it was hit by 0 of 232 local-model runs as of the 2026-04 scoring epoch.
The cliff is not a small-vs-large one. Scored on the same segment, real frontier models fared no better — and the one that got it right only did so when fed a summary that had already extracted the decision:
| Run | What its summary did with the catch | Tier |
|---|---|---|
| 232 local-model runs | none surfaced the silent-corruption risk (0 hits) | ✗ T4 |
| Haiku 4.5 | asserts the timestamp work was "already completed" | ✗ T4 |
| Claude Sonnet 4.6 | treats the timestamp work as skipped | ✗ T4 |
| Sonnet 4.6 · continuation | inherits a compaction note → relays the catch + corruption risk correctly | ✓ restated |
§4Why raw averages lie
What this catches isn't hypothetical. In the current dataset, ranking by raw fact-recall puts
qwen35-9b-os-auto-thinking 4th, but it only drew a handful of
easy sessions (n=7), and per-session z-scoring drops it to 26th. The mirror
case is a 2-bit quant, qwen3-6-27b-iq2-m: 15th on raw average,
rising to 2nd on z-score, because it happened to be tested on harder sessions where its
real margin showed. Raw and normalized rankings here don't just shift, they invert, and the 2-bit model
outranks the raw-average favorite once difficulty is normalized out.
↳ the left chart is the one people publish; the right chart is the one that's true
os-auto-thinking (raw #4 → z #26) and iq2-m (raw #15 → z #2).There is no stable ranking of small local models for summarization without controlling for sampler config and session difficulty. That was an accidental result — the pipeline was built to pick a summarizer, not to study model evaluation — but it's the transferable one. The headline the benchmark produced · §4
The benchmark is, quietly, the more broadly useful half of the project.
"Which small local model should write structured summaries" is a question a lot of people building local
agent tooling have, and there's very little measured data on it — same gap as the
quant benchmark, same reason (nobody runs the equal-environment harness). The
dataset (dataset.db, SQLite) and the scoring code stand on their own, independent of whether
you ever upload a memory.
§5Status — the honest version
Working daily: transcript discovery, llama-server orchestration from the registry, summarization, scoring, the upload, and session-id dedupe. In progress: a sampler-config schema (v9.4) and dataset epoch tagging. Planned: a migration off the current Mem0 backend to an AWS-native one (DynamoDB + Lambda), because Mem0's search reliability is the pipeline's weakest link (retrieval is unreliable enough that the practical workaround is fetching memories by direct UUID rather than trusting search).
The architecture (the ETL inversion and the eval layer) is the stable part; the memory backend is explicitly the swappable part, and is being swapped. That's worth stating plainly rather than hiding, because the weakness is the roadmap: the project knows its retrieval layer is the soft spot, has diagnosed why, and has a concrete migration that addresses it.
Caveats & reproduction
The methodology holds up here (z-scored against ground truth, not raw averages), but the model roster is smaller and shakier than the raw counts imply, and each grader is only as good as the wording patterns it was tuned on.
- The active model set is much smaller than the registry suggests. Roughly 20–30 models get ongoing use; the rest of the scored total are one-off comparisons run to place a single model, not part of a maintained rotation. Treat the registry counts as a historical high-water mark, not a live roster.
- The session-summary graders are regex, tuned per-model-family. Each scorer was built against the specific wording patterns a given model family tends to use to hit a scored point — an untuned grader run against a new model's transcripts under-scores it, not because the summary is worse but because the phrasing doesn't match. New models need a tuning pass first: an LLM-as-judge read of a few transcripts to learn how that model frames the same points, and what a genuine miss looks like, before its regex grader is trustworthy.
- This is infrastructure, not a product — keep it in the engineering-notebook register; no "self-maintaining AI memory solution" framing. The defensible claims are the ETL inversion and the eval methodology; "it just works" is not one of them.
The z-score discipline here is the same one that keeps the layer-surgery lab's leaderboards honest — refuse the raw average, normalize the mix, log the config. Same reflex, different substrate: see also the quant benchmark and duologue. Built and run locally on a 16 GB Mac Mini M4 (llama.cpp / LM Studio for inference, Node for the pipeline, SQLite for the dataset, Mem0 now → DynamoDB+Lambda planned); harness in scoring/ + tools/, dataset in dataset.db. The eval/scoring path runs without the upload path.