Domain-Routed Agents vs. Naive RAG: A Comparison on the Expert System Task for a Tabletop Role-Playing Game
Langfuse
The author of the project, inspired by the task of creating an expert system for the universe of Vampire: The Masquerade, compares two approaches: classic RAG and a multi-agent architecture with domain routing. For evaluation, they collected a dataset of 40 questions of varying difficulty levels. The results show that the agentic approach outperforms naive RAG in terms of answer completeness for cross-domain queries.
An author, a novice Storyteller in the tabletop role-playing game Vampire: The Masquerade, needed to quickly find rules and lore. He created an expert system based on the official wiki, scraping 176 files (~750K tokens) divided into 7 domains. Instead of classic RAG, he chose a multi-agent graph on LangGraph, where a Router directs queries to domain experts, who then read the needed files themselves. For evaluation, the author compiled a dataset of 40 questions (10 level 1 — fact retrieval, 10 level 2 — synthesis, 10 level 3 — cross-domain, 5 for hallucination defense, 5 out-of-domain). Two approaches were compared: naive vector RAG and a multi-agent agent. The agent showed better quality on cross-domain questions, although it lags behind RAG in speed and cost.
Source: Habr — хаб NLP —
original