From Broken BI to AI-Powered HTML Reports: How Dzen Integrated LLMs into A/B Test Analysis
Yandex DataLens
The technical analytics team at Dzen developed a custom HTML report system with an integrated LLM to analyze A/B test results, focusing on the impact on authors, publications, finances, and the recommendation system. The system solves the shortcomings of traditional BI tools and delivers automatic, unified insights.
The team at Dzen, led by Mikhail Ryazansky and Mark Khabarov, created a tool for A/B test analysis that includes LLM integration. Initially, they tried using a BI system (DataLens) but it failed to handle the data volume. They built lightweight HTML reports accessible via a messenger bot. The reports include: author-level analytics, top authors with LLM-generated insights, financial impact on authors, and impact on the recommendation system. In a second iteration, they added publication-level analytics and a final LLM summary page that evaluates the experiment against strategic goals. The architecture uses user logs, experiment dictionaries, aggregated data, and an internal LLM to generate reports. They emphasize the importance of understanding the full ecosystem impact of A/B tests beyond simple metric improvements.
- Сокращения
- LLM = Large Language Model — большая языковая модель
- BI = Business Intelligence — деловая аналитика
- DWH = Data Warehouse — хранилище данных
- ML = Machine Learning — машинное обучение
Source: Habr — хаб ML —
original