Testing Checklist for Search and RAG Systems: Six Levels of Checks
A checklist for testing search and RAG systems is presented, comprising six levels of checks—from basic functionality to handling incomplete data. Metrics, tools, and a glossary of over 50 terms are described. The importance of the sequence of levels is emphasized: it makes no sense to evaluate RAG if the zero level has not been passed.
A detailed checklist for testing search engines and RAG architectures has been published on Habr. The authors identify six levels of checks: Level 0 – functional (the engine retrieves by specified fields, corrects typos, handles empty queries and special characters); Level 1 – classic Information Retrieval with metrics like Precision@k, Recall@k, NDCG, MRR, MAP on a labeled golden query set; Level 2 – quality of retrieved context for RAG (Contextual Precision and Recall); Level 3 – quality of RAG generation (Faithfulness, Hallucination Rate, Answer Relevance, Completeness, citation accuracy); Level 4 – agency and knowledge graph navigation (multi-hop, tool correctness, task completion, plan adherence); Level 5 – handling incomplete and weakly related data (abstention, over-extrapolation, conflict between outdated and current data). For each level, recommended tools are provided: pytest, Hypothesis, Postman, k6 for Level 0; ranx, trec_eval for Level 1; deepeval, RAGAS, TruLens for Levels 2–4; custom G-Eval for Level 5. Special emphasis is placed on performing checks sequentially – it is pointless to evaluate RAG metrics if the search engine fails basic tests, such as not retrieving a document due to a typo.
Source: Habr — хаб NLP —
original