MarathonMemBench: A Novel Benchmark for Testing Your LLM Agent's Memory
Anthropic
MarathonMemBench is a new type of benchmark platform that tests the long-term memory of LLM agents through extended, natural conversations, avoiding the need for special agent modifications. It was created because existing memory benchmarks either require loading pre-recorded dialogues (off-policy) or demand additional agent capabilities like state snapshots or programmatic memory access. The platform is designed to be agent-agnostic, requiring only a chat interface, and allows for the evaluation of an agent's ability to recall and maintain facts over long conversations that exceed the context window.
The article introduces MarathonMemBench, a platform for benchmarking the long-term memory of LLM agents. The author argues that the finite context window of transformer-based models is a fundamental limitation, and agents must use external memory tools, like writing to files, to overcome it. Existing memory benchmarks are criticized for requiring agents to be specially adapted: off-policy benchmarks (e.g., LoCoMo, LongMemEval, MemoryAgentBench, BEAM) load pre-recorded dialogues, while on-policy ones (e.g., AMemGym, MEMPROBE, MemoryArena, STATE-Bench, MemGym, ClawMark) demand state snapshots, programmatic memory access, or operating within their own environment. MarathonMemBench instead runs as a platform where an LLM persona converses with the agent over hours, presenting facts, changing them, and finally testing recall. It only requires the agent to have a chat interface. The article outlines plans for future sections covering existing benchmarks, the platform's architecture, and results from open-source agents, which were surprisingly strong.
- Abbreviations
- LLM = Large Language Model — большая языковая модель
- SFT = Supervised Fine-Tuning — дообучение с учителем
- RLHF = Reinforcement Learning from Human Feedback — обучение с подкреплением на основе отзывов людей
Source: Habr — хаб ИИ —
original