How I shrunk a Russian embedder to 24 million parameters — and almost ruined it with one digit

The author developed a small Russian dense retriever called STRIZH for a local RAG system running on an AMD Strix Halo node with 128 GB unified memory. The node's iGPU is shared by the generative LLM, embedder, reranker, and periodic reindexing, so a lightweight retriever is needed to reduce compute contention. After a failed attempt at embedding regression using MSE loss, the author successfully trained a retrieval donor model with 12 layers using contrastive learning and then distilled it to 4 layers. STRIZH has 24.4 million parameters, vector size 384, mean pooling, no query/passage prefixes, and supports context up to 8192 tokens. On a local corpus, it achieved Recall@10 of 0.751 on a filtered subset and 0.800 on the full set. In co-residency benchmarks with Qwen3.6-35B-A3B, STRIZH caused only a 2% generation throughput drop versus 7% for bge-m3 during 200 queries/sec online load, and indexed 46 batches/sec versus 4.9. However, a mistake was discovered post-publication: the tokenizer config had model_max_length=256, which when enforced led to a Recall@10 drop from 0.589 to 0.448. The author notes that STRIZH is not intended to replace larger models like USER2-small or bge-m3, but to fill a niche for efficient dense retrieval under strict compute budgets.