How I built an anti-scam bot on dirty data: detector, typifier and pitfalls along the way

The project addresses the problem of scammers in crypto chat rooms who read public messages and then privately message users asking for help. The bot detects public help requests and responds with a warning and a link to genuine support. The first challenge was building a classifier without a labeled dataset. Initial regex-based data collection limited the model's ability to generalize to unseen phrasing. Active learning was used to iteratively add uncertain examples, growing the positive set from 830 to 1176. Near-duplicates were kept to improve robustness to typos and slang. The classification criterion shifted from 'help request' to 'vaccine appropriate' to avoid triggering on simple questions. Cross-validation on a changing dataset gave misleading metrics; a frozen golden set of 245 messages revealed true improvements, with recall rising from 0.50 to 0.78. A separate test on real traffic showed precision tunable from 0.74 to 0.87 depending on threshold. The typifier (problem type) was kept as a simple logistic regression to avoid overcomplication. The system uses a single multilingual embedding model for both the detector and RAG, a compromise that saved resources but may limit RAG recall. The architecture is a separate FastAPI service with SQLite, called by a production bot via fire-and-forget, with a shadow mode for safe testing.