§0The problem with writing your own rules from memory

The shape of the result: an offline, map-reduce trawl over the agent's own logs that surfaces recurring friction with confidence tiers, and promotes the real patterns into the rules it runs on next session. A self-improving memory loop, with a human holding the final pen.

§1Two corpora, because they catch different things

The trawl reads two independent records of the same sessions, because they fail differently:

• Model-pipeline summaries — local-LLM meta-audits of each session, written to a friction-focused schema (Competence & Clarifications · Mistakes & Overreach · Friction Points · Waste & Efficiency). These are opinionated: a small model flagging where the agent overreached or wasted effort.
• Claude Code compaction summaries — the native format the agent emits when it compacts its own context (Primary Request · Errors and fixes · All user messages · Pending). These are factual: the signal lives in the verbatim user messages and the errors-and-fixes list.

Reading both matters because the opinionated summaries surface friction the factual ones can't see (a model flags "the agent moved too fast" where the compaction record just lists what got done), while the factual ones anchor the opinionated ones (a friction claim is worth more when a verbatim user interruption backs it). interpretation Triangulation is the method.

§2Why map-reduce, and why map must not judge

A hundred-plus summaries is, worst case, north of 800k tokens: you cannot read it in one context, and more importantly you shouldn't, since aggregating findings across a giant context is exactly where long-context attention degrades. Many small readers, each given a clean batch, beat one model squinting at the whole pile. So the architecture is deliberately split:

• Map (per ~30-summary batch): extract a flat list of observations — {session, type, one-line description, evidence}. The map agent does not tier and makes no cross-session claims. Judgment-light, runs cheap (Haiku or a local model), batches in parallel.
• Reduce, itself split in two: a mechanical pass clusters semantically-similar observations and counts distinct sessions per cluster (no judgment); a judgment pass (human + agent, in conversation) assigns tiers, applies cross-model logic, and drafts the actual rules.

The non-obvious constraint is why map must not tier: a pattern that shows up once per batch across six batches looks single-incident to every map agent. Only the post-aggregation global view sees that it recurred. observation Tiering before aggregation structurally cannot work — so it's forbidden at the map step.

§3The confidence discipline (the part that's actually the point)

Frequency is not evidence. This is the crux, and where the project stops being a log-grep and starts being eval methodology.

The tiers, and the axes behind them:

• Tier A: a pattern in 2+ distinct sessions (deduped by session ID), not already covered, becomes a candidate rule.
• Tier B: a single incident, but consequential, gated on demonstrated impact (a real user interruption, a concrete catch, quantified waste), never a bare summarizer label. (Precedent: the "surface a plan before editing" rule came from two single high-impact incidents that never recurred; frequency alone would have missed it.)
• Evidence-base is a separate axis from severity: a one-session incident can be high-severity. A friction signal has four independent axes: frequency (how many sessions), severity (how costly when it fires), source (a real user action vs. a summarizer's assertion), and observer-reliability (does the reader carry a known bias?). Collapsing all four into a single frequency count is exactly the mistake the next part is about.

The defense: any Tier-A candidate that pattern-matches the known bias profile must be transcript-verified before promotion — frequency is disqualified as evidence for it. Meanwhile, user-action signals (a real interruption, an explicit correction) stay trustworthy regardless of which model surfaced them, because they're grounded in what the human actually did, not what the summarizer asserted. interpretation Knowing your instrument's bias and routing around it is the whole game.

§4The failure that taught the most

The Phase-1 reduce agent failed, and the failure is more instructive than a success would have been. Two things broke. First, the agent's Write was blocked by a permission scope it didn't have, so it was forced to dump its report as chat text instead of a file. Second, and this is the keeper: forced to emit the report inline, the model confabulated session IDs. It had correctly read all 24 observation files, but when it assembled the clustered output it invented the per-cluster UUID lists: of ~30 IDs it cited, exactly one existed in the real corpus; the rest resolved to nothing. observation

The lesson generalizes hard: an LLM must never hand-transcribe a unique identifier. Carrying 68 exact UUIDs verbatim through a clustering operation is precisely the long-context transcription task models fabricate under. The fix is architectural, not a better prompt:

# every observation gets a short stable tag
model clusters by tag        # never touches a UUID
deterministic script: tag → session-count   # resolved from ground truth

(Fixing the Write permission alone would not have fixed the confabulation: they were independent failures, and conflating them would have shipped a still-broken pipeline.) That's the kind of bug that only shows up when you actually run the thing and then check the output against ground truth instead of trusting it, the same discipline the whole eval line is built on, pointed at its own tooling.

§5Knowing your instruments

§6What actually got promoted

The trawl isn't theoretical. It produced a candidate ledger of evidence-backed rules, each with source sessions and a destination, and the strong ones landed in a rewritten, lean CLAUDE.md. A sample of what the record surfaced that intuition had missed:

• Diagnostic phrases aren't directives. "Check if running," "continue," "let's draft it" are ambient, not action requests — surfaced from four sessions where they triggered unwanted action (one "check if running" killed three processes; one "continue" launched an unrequested retrieval).
• External-tool-output trust gate. Three sessions where a scorer result or a diagnostic was accepted as fact about the world without flagging its data quality.
• Verify the premise before searching. Sessions where the agent hunted for a log string or symbol the user described but that didn't exist in the code — all-model agreement on that one.
• Grep before reading whole files.
• Structured-source-first: query the registry, don't shell-parse filenames.
• Surface documentation at natural pause points (the user's own stated pattern: "chronic under-documentation unless explicitly prompted").

The outcome is a behavioural-rules file written from evidence (what demonstrably caused friction across real sessions) rather than from what felt memorable. And the loop is closed: those rules load next session, so the agent that helped mine its own friction is the one that stops repeating it.

Why it's more than a log-grep

1. It's eval rigor applied reflexively. Ground-truth verification, confidence tiers with evidence-base separated from severity, two-tier filter-then-confirm, explicit instrument-bias modelling — the same toolkit the layer-surgery lab uses on models, turned on the agent's own behaviour.
2. It's honest about being human-in-the-loop. Agents produce reports; no agent edits the rules file autonomously. The judgment step is where the human's ground-truth context is irreplaceable, and pretending otherwise is how you ship a confident wrong rule.
3. It caught its own confounds. The frequency trap and the UUID-confabulation fix are the project noticing where it would have fooled itself, and routing around it.

Caveats & status

This is explicitly human-in-the-loop: decision-support, not an autonomous self-rewriting agent, because an LLM promoting its own behavioral rules unsupervised is how bias compounds. Phase 1 shipped single-summarizer, bias-screened; the multi-model panel that would cross-check it is designed but deferred.

• The summarizers are small local models — they editorialize and occasionally fabricate; the whole confidence discipline exists because the readers are imperfect.
• The feeder pipeline is paused, not retired — the mem0-processor that produces the summaries is on hold; sessions after its last run have no summaries yet, so extending the trawl today means reading raw transcripts directly. Either way the method is the durable artifact, not the running service.