MELT

2026-07-18 · page · source ↗ · source summary

Open-source benchmark harness for evaluating long-lived memory in AI agent systems. Tests whether memory survives correction, contradiction, decay, and consolidation — not just retrieval accuracy. Built by Shisa AI alongside their shisad memory system when they found no existing eval that tested how memory works over time. Its most important finding so far is that one "memory score" is never enough: the system that wins on lifecycle dynamics can lose catastrophically on long-conversation QA, and single-number aggregates hide the per-axis weaknesses that actually matter.

Architecture

MELT is a harness, not a dataset. It runs replayable suites of cases against any memory system through a pluggable SUT adapter contract:

Config → Runner → SUT Adapter → System Under Test
                ↓
         Suite / Fixture
                ↓
      Scoring + Methodology
                ↓
         JSON Report

Core pipeline (src/melt/, ~3,500 LOC, zero PyPI deps, Python 3.11+ stdlib only):

SUT adapter contract (sut/base.py): 12-operation Protocol over JSONL-over-stdio (shisad) or HTTP (memobase) — hello/hello_ack, metadata, reset, ingest, memory_write, consolidate, manage, tick, memory_export, query, answer, shutdown. Versioned contract (currently B2) with capability negotiation.

Key Techniques

Lifecycle assertion taxonomy. Eight assertions beyond "did it retrieve?" Each probes a specific temporal behavior: - stored_fact_matches — basic capture quality - superseded_fact_absent — correction replaced old data - contradiction_detected — conflicts flagged, not merged - historical_fact_retained — as-of query returns past state - decayed_below_threshold — stale info aged below threshold - core_memory_stable — durable facts survive full consolidation - negative_evidence_absent — distractors excluded from results - scope_isolated — out-of-scope facts don't leak - abstention_on_unknown — system declines when it should

Each assertion maps to one of the 13 lifecycle axes, giving per-axis pass rates that diagnose which temporal behaviors a memory system gets wrong.

Dual write surface testing. Scripted track uses explicit memory_write operations (keys, values, predicates, supersession chain). Agentic track uses raw ingest events followed by autonomous manage calls. Same axis tested through different code paths — a system might handle structured facts but fail at extracting them from conversation.

Temporal/as-of queries. Queries carry as_of_time parameters. A correction case writes old fact at T1, superseding fact at T2, queries at T3 with as_of_time = T1.5 — the old value must return while the new value must be absent from current queries. This is impossible to test with static retrieval benchmarks.

Fake oracle as satisfiability validator. The deterministic fake SUT implements full memory semantics and passes every behavioral case by construction. If the fake can't pass a case, the case is broken. Published results show fake at 1.000000 across all axes, shisad at 0.953571, Memobase at 0.446429 on the full lifecycle suite.

Memory state dual introspection. Export mode scores against a full SUT memory dump with token efficiency metrics. Black-box mode scores from probe query evidence. Without memory_export support, write-precision collapses to recall and dedupe rate is trivially 1.0 — the report makes this transparency explicit. Only the fake SUT currently supports export.

Conversation-level LoCoMo cases. Ingests each conversation once, asks all questions against the same state — vs the legacy one-question-per-case shape that re-ingests the same conversation 155-199 times. Reduces SUT churn by 10-20× for full runs.

Config-as-reproducibility-hash. Resolved config (defaults + file + CLI + env) is SHA-256 hashed. Two reports with the same config hash used identical settings. Every report records fixture hash, SUT identity, contract version, runner commit, seed, run count, top-k setting, preliminary/final status, and judge identity.

Design Decisions

Decision Why
Zero PyPI deps No install friction, no supply chain risk. Cost: reimplements TOML, HTTP, SQLite from stdlib. Python 3.11+ only (needs tomllib).
Config-first Every setting in TOML with exhaustive validation. Enables reproducible runs. The 720-line config validator is the cost of confidence.
Preliminary vs final status Single run or dev split = preliminary; held-out with 5+ runs = final. Prevents overclaiming from single-run results.
Built-in fixtures Smoke/mini/full/stress tiers built in. Run lifecycle evals with zero downloads. Cost: generated cases can't match real conversation diversity.
Top-k bypass warnings When top_k >= session count, report warns retrieval scores are for sanity, not competition. Honest but could be gamed.
Baselines as first-class Raw_verbatim, no_context, gold_evidence, full_transcript baselines run alongside SUT. Enables leakage detection. Cost: up to 4× answer calls for judged QA.
Per-case checkpoints Every case writes checkpoint after completion. Resume picks up where it left off. Essential for multi-hour LoCoMo runs (published: 379s for Memobase lifecycle full).
Separate answer/judge models Answer can be cheap (GPT-5.4 Mini, $0.75/M input), judge can be careful (GPT-5.4, $2.50/M input). Published LoCoMo: $4.46 for shisad, $8.92 for Memobase.
Deterministic scorers default Retrieval and token-F1 judging are free. LLM judge is opt-in. Makes CI/pre-commit eval runs cost zero.

Preliminary Results: One Score Is Never Enough

Leonard Lin is explicit that all published results are preliminary — single development runs, known caveats, not leaderboard claims. But they're the most informative memory benchmark results available because MELT refuses to collapse them into one tidy number.

Lifecycle full fixture (217 cases, lifecycle-v4):

System Lifecycle pass Retrieval R@3 Correction Contradiction Temporal Maintenance
ShisaD 95.4% 90.3% 100.0% 100.0% 23.5% 100.0%
Memobase 44.6% 13.4% 56.7% 13.3% 23.5% 13.3%

Same systems on LoCoMo judged QA (1,986 questions, GPT-5.4 Mini answer / GPT-5.4 judge):

System Judged QA score
Memobase 42.0%
ShisaD 6.2%

The ordering flips completely. ShisaD dominates on lifecycle semantics; Memobase dominates on long-conversation QA. Neither aggregate tells the full story. And the per-axis breakdown reveals temporal/as-of recall (23.5% for both) as a shared weakness that a single "pass rate" would hide entirely.

This is the core argument MELT makes: memory isn't one thing. It's write quality, correction, contradiction handling, temporal recall, maintenance, abstention, and retrieval hygiene — and a system can be excellent at some while terrible at others. Any benchmark that compresses these into one number is doing violence to the problem.

Lin: "Evals are hard, and agentic memory evals are still nascent. Memory spans many behaviors, making any single score easy to overread. MELT is our attempt to add nuance and tease apart those dimensions."

Key caveats on these runs: retrieval-bypass warning because top_k=3 matches canonical session count; neither adapter exposes full memory export, so write-policy metrics use weaker black-box proxies; LoCoMo has known dataset and judging problems.

Comparison Notes

MELT is the only benchmark that systematically tests temporal memory dynamics — correction, contradiction, as-of recall, decay, and maintenance — rather than static retrieval accuracy.

Agentic Context Management reaches the same conclusion from the vendor side: Maximem's arXiv paper reports 92.0% LongMemEval / 93.2% LoCoMo but explicitly disclaims that these benchmarks measure recall, not latency, token efficiency, or context-rot resistance — the same per-axis honesty MELT enforces structurally. Its residual errors concentrate in LongMemEval's multi-session category (75.2%), which the paper names as the "reasoning-sufficiency regime" — exactly the temporal, cross-session dynamics MELT's lifecycle axes are built to probe and static retrieval benchmarks cannot.

Related memory infrastructure that could be evaluated through MELT's SUT adapter contract: Sawtooth Memory, Mnemo, LocalAI (includes memory service), Claude-Mem, Lemmalog — LLM Memory as Program Analysis (whose retractions and validity intervals are a direct implementation of MELT's correction, contradiction, and as-of-recall axes).

Shisa's own Jouzu harness is the applied counterpart to this eval: its pi-vcc compaction plus vcc_recall retrieval implements, in a shipping daily-driver, the compaction-and-recall lifecycle MELT measures in the lab — the benchmark and its tool shipping from the same shop, with the harness's "scanning is not a guarantee of safety" humility matching MELT's "one score is never enough."

Tags: #tool #benchmark #ai-memory #evaluation

Sources: MELT - Memory Evaluation for Lifecycle Testing, MELT: Testing Long-Lived Memory for Agentic AI Last updated: 2026-07-18