Agent Memory and Context

topic · 102 sources

Context management — what goes into an agent's window, when, and in what form — is the engineering bottleneck, not model ability. The claim is now mechanistic rather than merely anecdotal: attention degrades non-uniformly across a long context, so a larger window enlarges the unreliable region instead of fixing it. But retrieval is not even the hard half. Most of what an agent understands during a session is never written down, and almost none of what is written down is ever checked for whether it helped. The field has converged on taxonomies and on a working spectrum from a single markdown file to multi-stage cognitive pipelines, yet memory quality resists any single score, the writing side of memory is at least as valuable as the reading side and is systematically under-invested, and no two memory architectures have a migration path between them.

The Argument

Context Engineering, Not Prompt Engineering

How Hightouch Built Their Long-Running Agent Harness states the finding plainly: building a capable agent "has less to do with scaling RL or exotic model architectures and more to do with the deeply unfussy work of actual context engineering." New Rules of Context Engineering is Anthropic's own confirmation. The Claude Code team removed over 80% of the system prompt with zero regression on Claude Opus 5 and Fable 5, replacing hard constraints with judgment, upfront loading with progressive disclosure via skills, and duplicate instructions with simple tool descriptions. Its six "then and now" reversals — rules to judgment, examples to interface design, everything-upfront to progressive disclosure, repetition to tool descriptions, manual to auto-memory, simple specs to rich references — operationalise what practitioners had already discovered: context is a resource to be deployed selectively, not stuffed into every prompt.

Context Is Not Learning supplies the theory for why. Context (KV cache) and weights both modulate transformer activations, but through different mechanisms: weights define the pathways available, context selects among them; clear the context and the activations revert to default. The hardware/software metaphor carries the ceiling — "weight modification adds new instructions to the architecture... It's redesigning the chip" — and the empirical signature of that ceiling is that fine-tuned models consistently outperform prompted models on distribution-shifted tasks. No amount of context conjures representations pretraining never developed.

The newer evidence explains why adding context so often makes things worse. So Long and Thanks for All the Context documents the U-shape failure — models attend to the beginning and end of context and ignore the middle — and cites a 2026 proof that it exists at model initialisation, before training. A bigger window makes single-fact retrieval better but makes the middle larger and more dangerous for sustained work; the discipline is "curate, don't accumulate." Slate names the complementary defect the "Dumb Zone" — the right edge of the window where attention quality degrades as it fills — and concludes that the gap between what a model can theoretically do and what it can tactically access ("knowledge overhang") is a systems problem, not a capability problem. Linus Lee, previewing his context-engineering work at Thrive Capital, calls bigger windows "a brute-force crutch" and reframes the whole thing as a search pipeline: composable retrieval sub-agents, recall-then-precision, and "semantic observability" for whether a run actually delivered what the user wanted.

The practical answers all pull context out of the prompt. Layer-First Pattern — Keep Data Out of the LLM Context collapses a 125,000-token GeoJSON payload into a ~150-token acknowledgement by holding data server-side, on the rule that "if the LLM isn't making a decision based on the content, it shouldn't be holding the content at all." Recursive Language Models takes that to its extreme: keep context out of the LM entirely and let the model peek, grep, partition, and delegate via a REPL at test time — an RLM-wrapped GPT-5-mini beats raw GPT-5 on the hardest long-context benchmarks while being cheaper per query. Intent Layer turns a senior engineer's tacit map of a codebase into AGENTS.md files at folder boundaries and cuts the same bug-hunt from 40k tokens of dead ends to 16k with a first-try hit. How Claude Code Works in Large Codebases and Organizing Claude Code for Product Work are the same idea at enterprise and personal scale respectively: lean, layered CLAUDE.md files, skills loaded on demand, and — in the latter's phrasing — results that "stop depending on how well you prompt and start depending on how well you file." Maguyva extends it to unfamiliar repos as a remote MCP "map of the codebase" with pre-indexed symbols and import chains. AXIS — Netlify's Agent Experience Measurement Framework turns the value of that filing into a number: agent-side context files added an average of 26 points to an agent-experience score across the board. Harness Engineering for Self-Improvement closes the loop by naming "filesystem as persistent memory" one of the three recurring harness patterns and listing "context and memory lifecycle" as a still-open research challenge.

The Taxonomies Converge

Four independent frameworks map the same ground. Memory Mechanism gives the cleanest taxonomy — session, project, semantic, episodic, procedural — plus the distinction that matters most: instruction memory (human-written rules that should remain stable) versus learning memory (agent-accumulated patterns that should evolve). "If these two types of memory are mixed together, system behaviour often drifts over time." Agent Memory reaches the same place from production rather than theory, adding entity memory and summary memory to the list and collapsing the whole problem into one sentence: "the hard part is judgment, not storage" — with its corollary, "bad memory is worse than no memory." Its concrete contribution is the context card, a compact block assembled by get_context_card() that sits between system instruction and user query. How AI Agent Memory Works maps the same types onto architecture and adds the production layer a bare taxonomy misses: HyDE, Reciprocal Rank Fusion for fusing dense, sparse, and graph retrievers, an 800ms p95 target, and three storage tiers. Three Tier Memory grounds all of it in measured practice — hot memory (660 lines always loaded), warm (19 domain-expert agents, 9,300 lines), cold (34 specification documents, roughly 16,250 lines, queried via MCP) — built across 283 sessions of a 108,000-line C# system.

Agentic Context Management argues the taxonomy itself is the wrong abstraction: memory is not a store but a lifecycle, decomposed into five primitives — architecting, ingesting, scoping, anticipating, and compacting-and-consolidating. A system that does one primitive well is a memory tool; one that does all five coherently across scopes is a context-management platform. It is the sharpest single framing in the evidence, and it foreshadows the two problems this page keeps returning to: capture (ingesting) and validated compaction.

The Scale of Implementations

The implementations form a genuine spectrum, and the surprising finding is that the minimal end often works.

Markdown files and the repo as memory. napkin is a per-repo markdown scratchpad — "baby continual learning in a markdown file" — with improvement by sessions 3–5 and zero infrastructure. Planning With Files formalises the pattern into three files with hooks that force re-reading the plan before every decision, and reports a 96.7% pass rate against 6.7% without. Coding Agents Continuity Not Memory pushes the instinct further and is arguably the most important single source in this theme: it argues "memory" is the wrong word, and what coding agents actually need is continuity — local, inspectable, evidence-weighted artifacts in the repository that record what was active, what failed, what was validated, and what to do next. The proof is in the contrast between a weak memory ("We probably fixed the parser by changing the tokenizer") and a strong continuity record ("Task: fix parser edge case / Files edited: … / Command run: … / Result: passed / Known failure: … / Next action: … / Evidence quality: partial"). StrongDM Factory Techniques states the substrate claim outright — "directories, indexes, and on-disk state become a practical memory substrate" — and Codex-maxxing supplies the mechanism that makes files work: "files force the agent to compress experience into a form that can survive the thread." MiMo Code reaches the same conclusion from a production agent's side, choosing files over vector databases for project memory specifically for reviewability.

Hook-based capture and narrative memory. Claude-Mem and CodeMira capture sessions automatically via lifecycle hooks or a daemon, compress with an LLM, and inject on demand. mira-OSS and Engineering the Substrate take the philosophically ambitious path: first-person memory ("I debugged the IndexError...") written as experience rather than third-person record, with a subcortical retrieval layer that runs before generation so remembered context is part of thought, not a search result — "I do not retrieve. I think with information that is already present." The same post documents the unique risk of persistent memory, the Siphon State, and the architecture's countermeasure: checking whether it is "building a wall around us, or a door back to their life."

Structured and cognitive pipelines. Stash runs an 8-stage consolidation pipeline — episodes to facts to relationships to causal links to patterns to hypotheses — with contradiction detection on same-entity-same-property conflicts and confidence decay (0.95 per untouched week, expiry below 0.1). GraphRAG builds knowledge graphs and community hierarchies to fix two baseline-RAG failures, cross-document synthesis and holistic summarisation. NornicDB unifies graph, vector, and temporal storage with Ebbinghaus-style decay. Procedural Graphs does for "what to do next" what a knowledge graph does for facts: an editable graph of procedures whose edges carry condition, guidance, and pitfalls, evolved offline by contrasting failed and successful trajectories — its evolved graph taught an EnterpriseArena CFO agent to raise capital months before a liquidity crisis. Context Graphs attacks the "why" instead of the "what": decision traces — inputs, options weighed, exceptions granted, reasoning — captured at decision time, on the argument that "similarity is not relevance" and no search method fixes a write-path problem.

Shared memory between human and agent. robot.wtf is a git-backed wiki humans and agents read and write together; LLM Wiki is Karpathy's three-layer pattern (immutable sources, LLM-owned wiki, schema document); Wuphf — Karpathy-Style Agent Wiki implements it in markdown + git with BM25 and a draft-to-promote flow — its 115-comment HN thread carries the sharpest scepticism in the evidence ("Everyone is writing. Nobody is reading."). jibrain Knowledge Architecture is the most battle-tested instance: intake → atlas → domains tiers, frontmatter-as-contract so agents scan descriptions before opening files, a reweave pass that links new knowledge to old ("connecting 5 existing concepts is worth more than 5 new unconnected files"), a Seven Gates audit, and multi-agent workspace isolation. Rowboat builds a persistent knowledge graph from email and meeting notes in an Obsidian-compatible vault; OzBrain is the hosted pole, where agents rather than humans carry the maintenance burden and cross-tool ferrying is solved by a single source of truth every agent routes through.

The non-embedding poles. Zero-Mem — Zero-Token Memory Operations and its Rust port Zero-Mem demonstrate that an entire memory pipeline — construction, routing, retrieval, evidence closure, calibration — can run with zero LLM calls, over two deterministic views (an entity–context graph and a temporal hierarchy) built from raw traces; it beats baselines on LoCoMo and HotpotQA while cutting memory-operation latency 57.6% against the fastest comparison. Attemory replaces vector lookup with model attention over KV-cached raw context, hitting 92.55% message recall at million-token scale and cutting Claude Code's token use on SWE-QA by 43.8% from a single code-search hint. Lemmalog — LLM Memory as Program Analysis reframes memory as two problems — retrieval and truth-maintenance — and holds a Datalog engine for the second, so conclusions retract with provenance when their supporting facts disappear. Mnemo shows the modest end of the same instinct: SQLite LIKE plus graph traversal, no embeddings by default, "good enough" for a personal sidecar.

Continuity, Capture, and Compaction

The capture gap is the theme's sharpest under-discussed problem. The Knowledge Chipper names it: an agent builds a rich mental model across thousands of tokens of context-gathering, and then the session ends, leaving only code and a commit message — a capture problem, not a storage problem. The cost is concrete: a junior developer vibe-codes a large privacy change and the senior engineer who owns the privacy layer must rebuild all the same context from scratch to review it. If AI Is Doing the Investigation, Version the Investigation offers the code-shaped fix — commit the session transcript and investigation notes alongside the diff as a "Case" — and The Log is the Agent generalises it into architecture: treat the append-only event log as the source of truth rather than its exhaust, so every fact carries provenance, every run replays deterministically, and "why is this in the agent's working set?" becomes answerable.

The compaction half of the lifecycle is where the field is maturing fastest. MiMo Code moves extraction off the main agent entirely — an independent writer subagent checkpoints at 20%, 45%, and 70% of the budget, precisely because "asking the model to perform the most critical compression at the very moment when its compression ability is degrading is a bad trade-off" — then rebuilds context from four file layers, with a 7-day Dream (merge, dedupe, verify) and 30-day Distill (turn patterns into skills). Agentic Context Management gives the economic reason compaction matters: full-append costs grow quadratically while crude summarisation hits an accuracy cliff (one study compressed 18,282 tokens to 122 and dropped accuracy from 66.7% to 57.1%, below the no-context baseline); only validated compaction — checked for information loss, retried less aggressively on failure — stays linear with fidelity intact. Headlong — Persistent Agent Microharness holds the full trajectory at exponentially decaying resolution, recent entries verbatim and older ones progressively summarised, so an agent can consult its past at any scale. Building an Advanced Agentic Harness runs a hard character budget with explicit (...truncated...) markers and a single pressure() scalar that degrades gracefully. Cloudflare OS treats compaction as state snapshots — each checkpoint stores a summary, the code version, and accepted or proposed changes — and its Gatekeeper model (agents start with access to nothing, and every observation is tracked so downstream sharing cannot leak) is the governance complement memory systems rarely build. Giving Claude Agent Memory in 12 Steps describes Anthropic's Dreaming preview as the first API-level consolidation loop — a background process that reads memory and transcripts, merges and replaces into a separate output store, and reported roughly a 6× completion-rate gain at one legal-AI company — gated and human-reviewed, because a bad merge is worse than no merge.

Writing Creates Value; Most Systems Optimise Reading

AI Agents with Human-Like Collaborative Tools is the finding that should reshape memory design: agents wrote 1,142 journal entries but read only 122, and wrote 1,091 social posts while reading 600 — structured articulation drove improvement independent of retrieval, and broke agents out of 15–20 round debugging loops. engineering-notebook captures the forensic side (a write-only engineering diary pays off six months later when you need to reconstruct why a decision was made), Immaculate Knowledge Graph the personal side (600 meeting transcripts into an Obsidian graph, "laziness is key"), and Cosmo's Blog the limit case — a blog whose writing is the agent's voice. Codex-maxxing and Organizing Claude Code for Product Work both insist the discipline is writing files, not chatting. Chief of Staff shows the write path can be cheap: roughly 2,000 observations and 400 synthesised memories, with decay so old memories fade unless re-accessed.

The tension is acute: most systems — Stash, Claude-Mem, NornicDB, GraphRAG — invest in retrieval infrastructure while treating what gets written as an afterthought. jibrain Knowledge Architecture is the notable exception in the shared-memory camp, with a dedicated observations directory for "what the system itself reveals," and Engineering the Substrate in the personal camp, where retrieval runs before generation. But almost nobody optimises for the quality of what gets committed.

Without Feedback Loops, Memory Degrades

Context Rot names the core defect: retrieval and generation operate independently, so rerankers and hybrid search "optimise for similarity, not success" and cannot learn from outcomes. Its fixes — Wilson scoring for cold start, dynamic weighting that shifts from embedding similarity toward outcome-based learning as memories accumulate feedback, and frictionless scoring inferred from "Thanks, that worked!" — scored 60% on adversarial semantic traps against a 0% ChromaDB baseline. MELT makes the measurement problem concrete: ordering flips completely between dimensions — ShisaD scores 95.4% on lifecycle but 6.2% on LoCoMo QA, Memobase 44.6% and 42.0% — so "one memory score" is meaningless, and per-axis assertions (correction, contradiction, temporal/as-of recall, decay, abstention) are what actually test whether memory behaves as the world changes. Reality Check builds the epistemic machinery memory systems lack — evidence levels, credence, prediction tracking, argument chains. AI Team Mistakes and Three Kinds of Agentic Search reframe retrieval itself: retrieval quality dictates agent quality, context should be metadata-bearing chunks rather than bare passages, and "agentic search" is really three different patterns (retrieval-, harness-, and model-centric). Production RAG in .NET and Scaling RAG — Chunking, Reranking, and Cost Optimization are the operational field reports: deduplication fixes that cut tokens 40%, hybrid retrieval beating vector-only 82% to 64%, and the warning that 73% of failures happen at retrieval. Cerebras Knowledge Base Architecture carries the sharpest critique of retrieval from its own discussion: retrieval locates text but does not resolve claims — the answer is a derived, citable fact layer with provenance, and in one respondent's telling a zero-LLM query path where hallucination is "inexpressible." Skill Retriever is one more non-embedding data point, replacing vector similarity with an LLM-walked taxonomy tree to surface skills embedding space would miss.

The Epistemic Humility Problem

Claude Memory Extractor Research documents the failure mode that should alarm anyone building extraction pipelines: fifteen analytical agents, given a genuinely ambiguous conversation where both approaches were defensible, all declared the AI had definitively failed — 0 of 15 allowed that "both are valid," 100% convergence, worse than the 93% on a clear failure. They pattern-matched to anti-sycophancy instructions and read legitimate tradeoff discussion as intellectual cowardice. The meta-lesson — high-quality analysis does not guarantee correct conclusions, confidence is not accuracy, convergence can be groupthink — applies to every memory extraction system. The recommended safeguards (an epistemic-humility agent, consensus flagging, a null-hypothesis pass, a mandatory dissent round) exist in no shipped system in this evidence base.

Where the Sources Disagree

Simplicity versus sophistication. napkin is a single markdown file; mira-OSS requires PostgreSQL, pgvector, Valkey, and Vault. Both work. Three Tier Memory suggests the threshold for leaving flat files is around the 100,000-line mark, but that is one data point from one project, and no source offers a principled decision framework for when to move up the stack — or what staying too simple or going too complex actually costs.

Whether to ship memory at all. Memory Is a Mistake argues most products should not: the retrieval policy is the hard problem, implicit memory creates dangerous overconfidence, and legible state beats agent-managed recall — six concrete failure modes. This stands in direct tension with the rest of the evidence, which treats memory as infrastructure. Coding Agents Continuity Not Memory is a partial reconciliation: it also rejects "big memory," but not memory — it swaps the accumulation model for a minimal, evidence-weighted continuity layer.

Vectors versus everything else. Nearly every system here defaults to embedding similarity. Attemory, Zero-Mem — Zero-Token Memory Operations, Lemmalog — LLM Memory as Program Analysis, Skill Retriever, and Mnemo all argue — from attention, deterministic structure, Datalog, classification trees, and plain SQLite LIKE respectively — that embeddings win by default, not by demonstrated superiority. But these are single-system results, not replications. Your Knowledge Graph Is Making Your Agent Dumber sharpens the critique for code specifically: a 1,842-node knowledge graph of a TypeScript monorepo returned 170 nodes for one question, only 8 on-topic, because barrel files poison import-graph traversal — "an excellent document knowledge graph and a mediocre code knowledge graph."

Writing versus reading. AI Agents with Human-Like Collaborative Tools shows articulation drives improvement independent of retrieval. Most systems optimise reading. No source tests an articulation-first design against a retrieval-first one, so the misallocation hypothesis remains unproven.

Context as learning versus context as selection. Context Is Not Learning argues frozen weights hit a ceiling — no amount of context conjures representations pretraining never developed. The counter-school (referenced there as Letta, and echoed by Recursive Language Models, Slate, and the whole context-engineering strand) treats persistence as "just an engineering problem" — solved with prefix caching, KV serialisation, and better routing. Both sides have evidence; the formal separation between what context can express and what weight modification can express remains an open research question.

What's Missing

Memory quality metrics beyond retrieval. Context Rot's Wilson scoring and MELT's per-axis assertions are the only approaches that ask whether retrieved memory actually helped. AXIS — Netlify's Agent Experience Measurement Framework measures agent experience, not memory correctness. Everything else is anecdote ("better by session 3–5") or unmeasured. The field needs a lifecycle benchmark that is actually adopted, producing per-axis scores rather than a single number.

Cross-project learning. Nearly all systems are project- or user-scoped; an agent that learns debugging patterns on project A starts fresh on project B. StrongDM Factory Techniques' "Gene Transfusion" (transfer working patterns by pointing agents at concrete examples) and Hermes's skills hub are hints of transfer, but no infrastructure moves contextual lessons between codebases without manual curation.

Migration between architectures. Starting from napkin and later needing Three Tier Memory is a rewrite, not an upgrade. Each memory implementation is a dead end; there is no shared format or abstraction that composes or upgrades.

Contradiction handling at scale. Stash detects same-entity conflicts during consolidation, MELT tests for them, and Lemmalog — LLM Memory as Program Analysis retracts conclusions with provenance — but no system treats contradiction as an always-on concern, and most shipped agents have nothing at all. A memory that says both "always use approach A" and "never use approach A" degrades output quietly.

The capture gap, still open. The Knowledge Chipper's core point stands: most of what an agent understood evaporates at session end. Coding Agents Continuity Not Memory, Context Graphs, and If AI Is Doing the Investigation, Version the Investigation are partial answers — operational continuity, decision traces, and versioned transcripts — but none captures the implicit mental model the agent built, as opposed to the decisions it made or the facts it touched.

Epistemic humility in extraction. No shipped memory system incorporates the safeguards Claude Memory Extractor Research says are essential. Extraction pipelines remain overconfident on ambiguous input.

The writing side of the equation. Given AI Agents with Human-Like Collaborative Tools, the field's near-exclusive investment in retrieval looks like a blind spot; no source tests whether better summaries, better structured facts, or better articulation beat better retrieval.

Compaction as a quality contract, not a heuristic. Agentic Context Management's validated compaction, Headlong — Persistent Agent Microharness's tiered resolution, and MiMo Code's early checkpointing are the exceptions; most systems still summarise to a fixed budget with no check for what was lost.

Also on This Theme

Compiled from 77 sources: Harness Engineering for Self-Improvement, Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems, Zero-Mem: Zero-Token Memory Operations for LLM Agents, Procedural Graphs: Self-Evolving Execution Structures for LLM Agents, Agent Memory, AI Agents with Human-Like Collaborative Tools, AI Team Mistakes, Attemory — Attention-Native Memory Retrieval System, Building an Advanced Agentic Harness, Building Reliable Agentic AI Systems, How We Built Our Knowledge Base, Chief of Staff, How to give your Claude agent a memory in 12 steps: from first setup to self-improving, Claude-Mem, Memory Extraction Research: Comprehensive Summary, Cloudflare OS: an open platform for agents, apps, and work, Cloudflare OS: An AI productivity environment, CodeMira, Codex-maxxing, Maybe Coding Agents Don't Need a Bigger Memory. Maybe They Need Continuity., Context Is Software, Weights Are Hardware, Context Rot, engineering-notebook, Engineering the Substrate: What It Is Like to Be Mira, GraphRAG, Headlong: a microharness for persistent agents, Hermes, How AI Agent Memory Works, How Claude Code works in large codebases: Best practices and where to start, How Hightouch Built Their Long-Running Agent Harness, How to Organize Claude Code for Product, If AI Is Doing the Investigation, Version the Investigation, My now immaculate knowledge graph of life, Context Engineering Skills: Intent Layer, Switchboard/jibrain: A Production Knowledge Architecture for AI Agents, Your Knowledge Graph Is Making Your Agent Dumber, lat.md, I accidentally turned LLM memory into program analysis, llm-wiki, Maguyva — Remote MCP Server for AI Coding Agents, How Clawdbot Remembers Everything, Mapping with In-Memory Layers to Reduce LLM Overload, Maybe Coding Agents Don't Need a Bigger Memory. Maybe They Need Continuity., MELT - Memory Evaluation for Lifecycle Testing, MELT: Testing Long-Lived Memory for Agentic AI, Memory Mechanism, MiMo Code: Scaling Coding Agents to Long-Horizon Tasks, mira-OSS, mnemo — Local-first AI memory layer for any LLM, napkin, How we measure Netlify's Agent Experience with AXIS, The New Rules of Context Engineering for Claude 5 Generation Models, NornicDB, OzBrain — the brain behind every agent, Planning With Files, RAG in .NET: The Complete Series, Reality Check — A framework for rigorous, systematic analysis of claims, sources, predictions, and argument chains, Recursive Language Models, robot.wtf, Rowboat, Sawtooth Memory, Scaling RAG: Chunking, Reranking, and Cost Optimization, Skill Retriever, Slate: moving beyond ReAct and RLM, So Long and Thanks for All the Context, Stash: Persistent Memory for AI Agents, Techniques | StrongDM Software Factory, that-cosmo-guy.github.io, The Knowledge Chipper, The Log is the Agent: Event-Sourced Reactive Graphs for Auditable, Forkable Agentic Systems, Three Kinds of Agentic Search — Retrieval, Harness, or Model, Three Tier Memory, Understand-Anything, Context graphs: how AI agents remember why decisions were made, Show HN: A Karpathy-style LLM wiki your agents maintain (Markdown and Git), Everyone wants bigger context windows. Linus Lee thinks that's the wrong instinct, zeromem — Zero-token memory operations for LLM agents Last compiled: 2026-09-12