Agentic RAG, TRuST Benchmark and What Nafanya Has to Do with It
DeepSeek
Nafanya is an agentic RAG assistant for construction and pricing documentation. The author adapted the TRuST benchmark to test it, achieving improvement in metrics through architectural changes.
Nafanya is an agentic RAG assistant for normative and technical documentation in construction and pricing, developed by Nikolai. It uses a multi-agent architecture with a planner, retriever, evaluator, generator, and critic. To test its search capabilities, the author adapted the TRuST benchmark, which contains 324 complex Russian-language questions requiring multi-step search. Initially, the system scored low, but by introducing a CoverageCritic and optimizing the architecture, Recall All improved from 0.25 to 0.45 and answer correctness from 34% to 56%. Later, a Final Critic was added, and the system uses external APIs like Qwen and DeepSeek. The author notes that TRuST is based on the BrowseComp-Plus methodology and includes fixed index and various search complexity types.
- Abbreviations
- RAG = Retrieval-Augmented Generation — генерация с дополнением поиском
- LLM = Large Language Model — большая языковая модель
- VLM = Vision Language Model — мультимодальная модель
- API = Application Programming Interface — программный интерфейс приложения
- FTS = Full-Text Search — полнотекстовый поиск
- PDF = Portable Document Format — переносимый формат документов
- DOC = Document (Microsoft Word format) — документ Microsoft Word
- DOCX = Document XML (Microsoft Word format) — документ XML (формат Microsoft Word)
- TXT = Text file — текстовый файл
- HTML = HyperText Markup Language — язык гипертекстовой разметки
- TRuST = T-Tech Russian Search Test — тест русского поиска T-Tech
Source: Habr — хаб ИИ —
original