RUMBA: A New Benchmark for Evaluating Long-term Memory of Dialogue Systems in Russian

RUMBA (Russian User Memory Benchmark) is a new Russian-language benchmark for evaluating the long-term memory of dialogue systems, developed by the authors of the article. The dataset contains 85 original Russian-language dialogues between a user and an assistant, spanning an average of 191 days of communication (from 12 to 85 sessions, from 180 to 998 turns). In total, the benchmark includes 1,543 questions, each with a timestamp, which allows checking the relevance of facts at the time of the question. The questions are divided into three supergroups: Information Extraction (searching for and correctly using facts considering updates, deletions, and temporal dependencies), Reasoning (reasoning with comparison, calculations, causal relationships, and temporal context), and Abstention (the ability to acknowledge the absence of information). Each question has a multi-layer taxonomy: semantic type (17 types, e.g., StaticUser, UpdatingInfo, SocialRelationship, Arithmetic, MultiStep), number of evidence sessions (single- or multi-session), temporality (temporal or atemporal), and type of temporal expression (explicit, implicit, no temporal expression). The dialogues were created manually: user utterances were written by humans, assistant responses were generated using GigaChat Max, and questions and reference answers were manually verified. Twenty-six contributors participated in the collection, and the process took 20 working days. The English version was obtained by automatic translation and manually verified for cross-lingual comparison.