How I Failed to Make Gemma 4 Write Good Russian Poetry: A Cautionary Tale of Overfitting and Blind Metrics

Daniil Sheremet, an AI engineer at Agentic Lab, spent a month trying to get Gemma 4 to write Russian poetry with proper iambic meter and rhyme. His plan involved fine-tuning with LoRA, a critic-revisor system, constrained decoding, and other techniques. He built a custom poetry scanner inspired by Ilya Kozyrev's RPST tool to evaluate meter and rhyme. He curated a dataset from ArsPoetica, creating 13,342 training pairs with control tags. The first LoRA run (v1) produced gibberish words like 'нагосты' and 'гу́нты' due to overfitting and the use of stress marks in training targets that fragmented tokens. The second run (v2) with clean targets and lower learning rate produced more coherent but still flawed verse, and an A/B test suggested it was better than the base model. However, the evaluation grader turned out to be blind: it couldn't distinguish poetry from prose, and all previous rankings were noise. After fixing the grader with a neural accentator (RUAccent), better rhyme detection, and reweighted metrics, the base Gemma 4 model beat both fine-tuned versions on average and best-of-4 scores. The fine-tuned models optimized meter at the expense of rhyme, learning exactly what the tags emphasized. The story highlights the importance of robust evaluation and avoiding exotic token spaces in fine-tuning.