How to Translate Mixed Russian-Kazakh and Not Go Crazy
Researchers from MWS AI and universities proposed a method for generating a synthetic dataset for machine translation of mixed Russian-Kazakh text (code-switching). Their model, trained on synthetic data, outperformed commercial systems in human evaluation, though it lagged behind them in automatic metrics.
In Kazakhstan, people often mix Kazakh and Russian in social media and everyday speech (code-switching), but there are almost no machine translation systems that can handle such mixed phrases well due to a lack of data. Researchers from MWS AI and several universities collected a corpus of 618 code-switched sentences for evaluation, but this is insufficient for training. So they created a synthetic dataset using existing parallel corpora (legal documents and press releases) and the SimAlign method, which replaces Kazakh words with Russian ones based on context. This approach (cs-5) proved more effective than naive substitution. They fine-tuned mBART, M2M100, NLLB-600, and NLLB-3.3B models on synthetic data, and also trained a transformer-600 from scratch. By automatic metrics (BLEU, ChrF++, COMET), commercial translation systems outperformed all open models. However, in manual evaluation by native speakers using a Likert scale (1 to 5), the authors' best model scored 3.09, while commercial systems scored 2.80 and 3.49. Additionally, adding Russian-language tweets translated into Kazakh from the Rybakova corpus provided a slight quality improvement. The authors acknowledge limitations: synthetic data is imperfect (average naturalness score 2.62/5), the test corpus is small (618 sentences), automatic metrics do not match manual evaluation, comparison with commercial APIs is invalid (their models are unknown), and transferability to other languages has not been tested.
Source: Habr — хаб ML —
original