External Causal Recall Benchmark: Testing AI Assistant Memory on YDB from Yandex

The authors built a causal recall benchmark on the public ydb-platform/ydb repository (10,285 issues, 37,067 PRs, C++, English). They extracted gold pairs using the GitHub Closes/Fixes/Resolves #N pattern (293 valid pairs after filtering out bot PRs from ydbot and github-actions). For each query (issue title), two retrieval methods were compared: (a) semantic cosine similarity top-32 using multilingual MiniLM embeddings, and (b) traversal of causal edges from the same seeds up to depth 3. Semantic recall was 32.1% (94/293), causal recall was 94.5% (277/293), yielding a graph contribution of +62.5 percentage points. The authors stress that the key result is not the exact numbers but the reproducible pattern: semantic word similarity alone maxes out around one-third, while explicit causal links dramatically boost recall. They provide full gold pairs and a 6-step method to allow independent replication. The work also notes incidental findings: 143 PRs created by AI agents (mainly GitHub Copilot) and 173 revert PRs, illustrating that memory problems are real.