RAG: Measurements Matter More Than Code. The Story of a Bot Where 'Correct' Improvements Worsened Results

A developer decided to build a RAG bot for targeted support of crypto chat users: the bot should find similar queries and show ready-made admin responses. On a corpus of 3.9 million messages and an evaluation set of 51 questions, he measured a baseline hit@3 of pure vector search at 74%. When trying to improve the system, he added hybrid search with BM25 and a cross-encoder reranker. Both changes dropped the metric: hybrid gave 55%, reranker 64%, hybrid+reranker 57%. Analysis showed that BM25 is ineffective due to the lack of exact tokens in the sample question corpus, and the reranker spoils already good ranking. The only beneficial change was tuning the cutoff threshold for the bot's role (a safety net, not a final answer provider): this improved useful output without requiring additional code. The main takeaway is that in RAG, what matters are not industry defaults but honest measurements and data cleaning.