Fine-tuning ruGPT-3 XL 1.3B to Modern LLM Level and Comparison with Gemma3 1B

The article's author fine-tuned an old pre-trained model, ruGPT-3 XL 1.3B, released in 2020-2021, which originally could only continue text. Using Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) methods, he transformed it into an instruction-oriented Large Language Model (LLM). For SFT, the dataset IlyaGusev/saiga_preferences (30k records) was used; training took about 1.5 hours per epoch using SFTTrainer from Hugging Face. After fine-tuning, the model learned to understand questions, provide correct answers, follow a dialogue template, and end generation with an End Of Sequence (EOS) token. The author also performed a comparison with the modern model Gemma3 1B (2025), trained on 2 trillion tokens. In tests on knowledge of Russian rock bands, multi-turn dialogues, complex physics questions, riddles, and concise answers, ruGPT-3 XL showed better or comparable results, although in some tasks, such as emulating a Linux terminal, Gemma3 1B performed better. It is also noted that the model can use a Chain of Thought (CoT) to improve response quality. After fine-tuning, the model was converted to the GGUF format for convenient use in llama.cpp.