When the Signature Is Not Enough: Clearing the 'Gray Zone' in Image Search
Яндекс
Yandex Images improved relevance by using multimodal models that jointly analyze image and text. They started with a heavy VLM, distilled it into lighter models for each pipeline stage, and increased relevant images in the top results by 5%.
Yandex Images historically ranked documents using two separate signals: image-to-text (i2t) and text-to-text (t2t) relevance, but this approach struggled in the 'gray zone' where these signals disagreed. The team, led by Konstantin Nikolaev, first built a multimodal VLM that processes image and text together, but it was too slow for real-time ranking. They distilled it into a set of lighter models: a bidirectional encoder called vBERT, a bi-encoder, an embedder, and a light cross-encoder (LCE). The embedder is used for candidate generation with an HNSW index, the bi-encoder for draft ranking, and the LCE for final reranking. These models were added as an additional factor to the existing ensemble, not replacing it. The changes resulted in a 4% increase in relevant images at the top from the bi-encoder and cross-encoder, and 5% total including the embedder. The key insight was to compute document embeddings offline and only compute query-side and aggregation at runtime, enabling real-time processing with tens of thousands of queries per second.
- Abbreviations
- VLM = Vision-Language Model — мультимодальная модель видения и языка
- i2t = image-to-text — изображение-текст
- t2t = text-to-text — текст-текст
- MLP = Multilayer Perceptron — многослойный перцептрон
- ANN = Approximate Nearest Neighbor — Приближенный поиск ближайшего соседа
- HNSW = Hierarchical Navigable Small World — иерархический граф малого мира
- LCE = Light Cross-Encoder — лёгкий кросс-энкодер
- GRPO = Group Relative Policy Optimization — групповая оптимизация политики
- DPO = Direct Preference Optimization — прямая оптимизация предпочтений
- DCN = Deep & Cross Network — глубокая и перекрестная сеть
Source: Habr — хаб ML —
original