RAG: Benchmarks Matter More Than Code. The Story of a Bot Where 'Proper' Improvements Worsened the Result

A developer set out to build a RAG bot for targeted support of cryptocurrency chat users: the bot was to find similar queries and show pre-existing admin responses. Using a corpus of 3.9 million messages and an evaluation set of 51 questions, he measured a baseline hit@3 for pure vector search at 74%. Attempting to improve the system, he added hybrid search with BM25 and a cross-encoder reranker. Both changes lowered the metric: hybrid gave 55%, reranker 64%, and hybrid+reranker 57%. Analysis showed that BM25 was ineffective due to the absence of exact tokens in the sample question corpus, while the reranker spoiled an already good ranking. The only beneficial change was adjusting the cutoff threshold to fit the bot's role (as a fallback, not a final answerer): this improved useful output without requiring additional code. The main takeaway is that in RAG, what matters is not industry defaults, but honest measurements and data cleaning.