Domain-routed agents vs Naive RAG: A Comparison on an Expert System Task for a Tabletop Role-Playing Game
Langfuse
The article describes the development of an expert system for the tabletop RPG Vampire: The Masquerade, comparing a domain-routed multi-agent approach with naive vector RAG. The authors collected a knowledge base of 176 files (~750K tokens) from an official wiki, categorized into 7 domains. They built a LangGraph-based agent with a Router, parallel domain Experts, and a Synthesizer, and tested it on 40 questions of varying complexity. The agent outperformed naive RAG in retrieving cross-domain facts and handling out-of-scope queries.
The author, a first-time Storyteller for Vampire: The Masquerade, faced the challenge of memorizing hundreds of pages of rules. To assist, they built an expert system using an official community wiki as the knowledge base. The wiki was parsed via Playwright, cleaned, and split into 7 domains: Mechanics, Disciplines, Lore, Clans, Sects, Skills, and Attributes, totaling 176 markdown files (~750K tokens). Instead of naive RAG, which fails on cross-domain queries due to chunking and semantic distance, the author implemented a multi-agent LangGraph architecture. The system consists of a Blackboard (shared state), a Router that identifies relevant domains, parallel Expert agents that read only necessary documents via a read_document tool, and a Synthesizer that produces the final answer. For evaluation, 40 questions were created: 10 factual, 10 synthesis, 10 cross-domain, 5 hallucination tests (old lore), and 5 out-of-domain. A checklist of required facts was made for each question. The system was compared against a naive vector RAG baseline using Langfuse for logging and scoring.
- Сокращения
- RAG = Retrieval-Augmented Generation — генерация с дополнением извлечением
- LLM = Large Language Model — большая языковая модель
Source: Habr — хаб NLP —
original