LLM-wiki vs. RAG: Comparison of Answer Generation Methods for Corporate Documents

The author conducted an experiment comparing two approaches to generating document-based answers: LLM-wiki (inspired by Andrej Karpathy's idea) and a simple RAG system with a vector retriever. The source text was a synthetic corporate document titled "Directive 401" about 100,000 characters long, prepared by the gemma4:31b model. Questions (20 factual and 20 case-based) were synthesized by the gpt-5.1 model. To build the wiki, the source document was split into 25 parts (24 articles plus a title page), then the file AGENTS.md was added in Obsidian, and using Claude Code (Sonnet 5), 32 wiki pages were generated in 20 minutes at a cost of less than $3.7. The wiki agent consisted of four components: Navigator (gpt-4.1-mini), Linker (gpt-4.1-mini), Context Collector, and Synthesizer (gpt-4.1). The RAG system used LangChain, ChromaDB, FastAPI, and local embeddings; chunks were created based on the second level of article numbering (144 chunks total). The evaluation used Precision and Recall metrics: facts from the answers were compared against a reference—the answer from an LLM that had seen the entire text. Results from three wiki agent configurations (no access to source, with access, with access and linker) showed that the median recall for LLM-wiki was about 0.72 compared to 0.54–0.57 for RAG, with p-values from 10⁻⁸ to 10⁻¹¹ using the Wilcoxon test. For precision, the difference was statistically significant only in two configurations (p < 0.05) and disappeared entirely when the linker was added (p = 0.348). The linker did not provide a statistically significant improvement. Access to the source moderately helped on fact questions, but the effect was marginal. The author notes that conclusions are limited to this dataset.