The viral claim: raw verbatim storage beats everything for AI memory. We reproduced the benchmark and found a different answer. The bottleneck is embedding truncation, not extraction loss. Here is the data.
Why We Ran This Benchmark
MemPalace (github.com/milla-jovovich/mempalace) launched on April 5, 2026 and reached 14,500 GitHub stars in 48 hours. The core thesis: store every conversation verbatim in ChromaDB, skip LLM extraction entirely, and you get 96.6% recall on the standard AI memory benchmark (LongMemEval).
The implication for the field: every system that uses an LLM to extract or summarize memories is over-engineering the problem. Raw text with good embeddings wins.
We build knowledge management systems based on the SECI model (Nonaka & Takeuchi, 1995), where structured extraction is a core operation. If raw storage genuinely beats extraction, our architecture is wrong. So we tested it.
What We Tested
Benchmark: LongMemEval_S (Wu et al., ICLR 2025). 500 questions testing five long-term memory abilities: information extraction, multi-session reasoning, knowledge updates, temporal reasoning, and abstention. Each question includes ~48 "haystack" conversation sessions, of which 1-3 contain the answer.
Four retrieval approaches, same embedding model (all-MiniLM-L6-v2), same data:
| # | Approach | Description |
|---|---|---|
| 1 | Raw all turns | Store full session text (user + assistant), embed as-is |
| 2 | Raw user-only | MemPalace's method: store only user turns, embed as-is |
| 3 | SECI extraction | Condense each session into ~500 chars of structured facts |
| 4 | SECI hybrid + keyword | Fuse raw + extracted scores via reciprocal rank fusion, add keyword overlap boost |
All approaches use ChromaDB with cosine similarity search. No LLM involved in retrieval. The only difference is what gets indexed.
Results
LongMemEval_S: Session-Level Retrieval (500 questions, all 6 types)
| Approach | R@5 | R@10 | NDCG@5 | NDCG@10 |
|---|---|---|---|---|
| Raw all turns | 85.9% | 92.8% | 80.3% | 83.0% |
| Raw user-only (MemPalace method) | 92.1% | 96.3% | 86.8% | 88.5% |
| SECI extraction | 93.7% | 96.7% | 89.1% | 90.3% |
| SECI hybrid + keyword | 93.9% | 96.6% | 89.7% | 90.8% |
SECI hybrid beat MemPalace-style raw storage by 1.8 percentage points on R@5.
Per-Question-Type Breakdown
| Approach | Know. Update (n=72) | Multi-Session (n=121) | SS-User (n=64) | Temporal (n=127) |
|---|---|---|---|---|
| Raw user-only (MemPalace) | 98.6% | 90.6% | 92.2% | 86.9% |
| SECI extraction | 95.8% | 93.2% | 96.9% | 88.8% |
| SECI hybrid+kw | 96.5% | 93.7% | 96.9% | 88.4% |
SECI extraction leads on 4 of 6 question types. MemPalace method wins on knowledge-update (+2.1pp), where original wording of changed facts matters. The SECI advantage is strongest on single-session-user questions (+4.7pp) where the answer is buried deep in a long conversation.
Why Extraction Wins: The 256-Token Truncation Problem
The result surprised us. MemPalace's thesis is intuitive: extraction loses information, raw preserves everything. In theory, raw should win.
The margin is tighter than our initial 100-question sample suggested (+4.9pp narrowed to +1.8pp at full 500-question scale). Temporal reasoning and knowledge-update questions brought the gap closer. That is exactly why you run the full benchmark.
The reality: the embedding model cannot read the full document.
all-MiniLM-L6-v2 (ChromaDB's default, used by both MemPalace and our benchmark) has a maximum sequence length of 256 tokens. That is roughly 1,000 characters.
LongMemEval sessions average 10,000 characters. Some exceed 30,000.
When you store a raw session and embed it, the model reads the first ~1,000 characters and ignores the rest. 90% of the content is invisible to retrieval.
Our SECI extraction condenses the entire session into ~500 characters of structured key facts. Every extracted fact fits within the 256-token window. Nothing gets truncated.
The "raw always wins" thesis holds only when your embedding model can actually read the full document. At 256 tokens, it cannot.
Raw session (10,000 chars):
[████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]
↑ embedded (1,000 chars) ↑ truncated (9,000 chars) ← invisible to search
SECI extraction (500 chars):
[████████████████████]
↑ entire content embedded ← nothing lost
Would a Longer-Context Embedding Model Change the Result?
Likely yes. Models like bge-large-en-v1.5 (512 tokens) or nomic-embed-text-v1.5 (8192 tokens) would let raw storage embed more of each session. MemPalace's published 96.6% may use optimizations beyond the default model. We are testing longer-context embeddings next.
The point stands: your retrieval quality is bounded by your embedding window, and most developers do not check this.
What MemPalace Got Right
Credit where due:
-
User-only indexing. Stripping assistant turns improved raw retrieval from 85.9% to 92.1%, a 6.2-point jump. Assistant responses add noise (generic, verbose) that dilutes the embedding. Smart design choice.
-
Knowledge-update performance. Raw user-only scores 98.6% on knowledge-update questions versus our 96.5%. When facts change over time, having the original wording helps. Extraction can smooth over the update signal.
-
Benchmarking culture. Publishing reproducible LongMemEval results with scripts, being transparent about what mode (raw vs AAAK vs rooms) produces which score, and issuing honest corrections within 48 hours of launch. That is how open source should work.
-
The core insight is partially correct. At scale with the right embedding model, raw storage is a strong baseline. The field has been over-engineering extraction with expensive LLM calls when simpler approaches work. The nuance: "simpler" must account for the embedding window.
The SECI Model for AI Memory
Our extraction approach is based on the SECI knowledge management model (Nonaka & Takeuchi, 1995), adapted for AI agent memory:
| Phase | Knowledge Flow | Implementation |
|---|---|---|
| Socialization | Tacit → Tacit | Session happens, context is experienced |
| Externalization | Tacit → Explicit | /distill extracts structured facts into markdown |
| Combination | Explicit → Explicit | /consolidate merges, deduplicates, detects decay |
| Internalization | Explicit → Tacit | /remember loads relevant context into next session |
The system stores extracted knowledge as flat markdown files in 10 namespaces (brain, patterns, solutions, research, content, voice, clients, projects, docs, quick-reference). Each file is human-readable and version-controlled.
For this benchmark, we added ChromaDB underneath the markdown files: dual indexing of both raw text and extracted summaries, with reciprocal rank fusion at query time.
Methodology
Dataset: LongMemEval_S (Wu et al., "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory," ICLR 2025). 500 questions, ~48 haystack sessions per question, mean session length 10,042 characters.
Embedding model: all-MiniLM-L6-v2 (384-dim, 256-token max sequence length). ChromaDB default.
Retrieval: Session-level granularity. Top-10 retrieval. Metrics: Recall@5, Recall@10, NDCG@5, NDCG@10.
Extraction method: Rule-based condensation (no LLM). Extracts topic from first user message, follow-up from last user message, resolution from last assistant message. ~500 characters per session. This simulates our /distill command's output structure.
Hybrid fusion: Reciprocal Rank Fusion (k=60) of raw and extracted retrieval scores, plus keyword overlap boost (up to 30% for exact term matches).
Sample: 500 questions (470 evaluated, 30 abstention skipped). All 6 question types: single-session-user, single-session-assistant, single-session-preference, multi-session, temporal-reasoning, knowledge-update.
Code: seci_vs_mempalace.py
Caveats
- Top-10 retrieval. MemPalace benchmarks with top-50 retrieval (n_results=50) and evaluates R@5 from that pool. Our top-10 is more constrained. Fair comparison with matched parameters forthcoming.
- Rule-based extraction. Our extraction uses regex/heuristic condensation, not LLM summarization. LLM-based extraction would likely improve quality but add cost and latency.
- Single embedding model. Results are specific to all-MiniLM-L6-v2 (256 tokens). Longer-context models would narrow the gap between raw and extracted approaches.
- Knowledge-update is a weak spot. MemPalace method scores 98.6% vs our 96.5% on questions where facts change over time. Extraction can smooth over the update signal.
What This Means for Builders
If you are building an AI memory system:
-
Check your embedding window. If your embedding model truncates at 256 or 512 tokens and your documents are longer, you are losing information at the embedding layer regardless of your storage strategy.
-
Extraction is not always lossy. When extraction condenses a long document into a representation that fits the embedding window, it preserves more retrievable information than raw storage that gets truncated.
-
User-only indexing helps. Stripping assistant turns from conversation memory reduces noise. A 6.2-point improvement for free.
-
Hybrid retrieval adds ranking quality. Fusing raw and extracted scores via RRF does not improve recall over extraction alone, but it improves NDCG (ranking quality). The right documents rank higher when you combine both signals.
-
Benchmark your system. LongMemEval is free, well-designed, and reproducible. Running it takes a few hours. The results may surprise you.
Reproducing These Results
# Install dependencies
pip install chromadb sentence-transformers
# Download LongMemEval data
mkdir -p data && cd data
curl -L -o longmemeval_s_cleaned.json \
https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json
cd ..
# Run benchmark
python seci_vs_mempalace.py --dataset s --mode retrieval
Full code: github.com/rankfor/ai-memory-benchmark
What We Are Testing Next
- Longer-context embedding models (bge-large, nomic-embed-text) to see if raw storage catches up when truncation is removed.
- LLM-based extraction (Gemini Flash for session summarization) vs our rule-based extraction.
- Temporal boosting for date-aware retrieval on the temporal-reasoning question category.
- Top-50 retrieval to match MemPalace's exact setup for a fair comparison.
The code is open. We welcome reproductions.
Dmitrij Żatuchin, Rankfor.AI. April 2026.
This research is independent. We have no affiliation with MemPalace, LongMemEval, or ChromaDB. The benchmark code and results are open for reproduction.