Testing Checklist for Search and RAG Systems: Six Levels of Checks
A checklist for testing search and RAG (Retrieval-Augmented Generation) systems is presented, covering six levels of checks — from basic functionality to handling incomplete data. Metrics, tools, and a glossary of over 50 terms are described. The importance of the level sequence is emphasized: there is no point in evaluating RAG if the zero level has not been passed.
A detailed checklist for testing search engines and RAG architectures has been published on Habr. The authors identify six levels of checks: Level 0 — functional (the engine retrieves by specified fields, corrects typos, handles empty queries and special characters); Level 1 — classic Information Retrieval with metrics Precision@k, Recall@k, NDCG, MRR, MAP on a labeled golden query set; Level 2 — quality of retrieved context for RAG (Contextual Precision and Recall); Level 3 — quality of RAG generation (Faithfulness, Hallucination Rate, Answer Relevance, Completeness, citation accuracy); Level 4 — agency and navigation through knowledge graphs (multi-hop, tool correctness, task completion, plan adherence); Level 5 — handling incomplete and loosely related data (abstention, over-extrapolation, conflict between outdated and current data). For each level, recommended tools are provided: pytest, Hypothesis, Postman, k6 for Level 0; ranx, trec_eval for Level 1; deepeval, RAGAS, TruLens for Levels 2–4; custom G-Eval for Level 5. Special emphasis is placed on the need for sequential testing — there is no point in evaluating RAG metrics if the search engine fails basic tests, such as not finding a document due to a typo.
Source: Habr — хаб NLP —
original