Alibaba Releases Qwen2.5-Coder: Up to 32B Parameters, Open Source Code Models

Alibaba's Qwen team released Qwen2.5-Coder, the successor to CodeQwen1.5, as part of the Qwen2.5 series. The models are available in three sizes: 1.5B, 7B, and a 32B version coming soon. They are trained on 5.5 trillion tokens, including source code, text-code grounding data, and synthetic data, with additional mathematics and general data to preserve non-coding abilities. The base model supports up to 128K tokens context, 92 programming languages, and achieves state-of-the-art results on code generation, completion, repair, and multi-language benchmarks. The 7B version outperforms DeepSeek-Coder-V2-Lite and CodeStral-22B on code tasks, and also shows competitive math performance on GSM8K and Math evaluations. The instruction-tuned variant, Qwen2.5-Coder-Instruct, further improves task performance and generalization, excelling in multi-language code reasoning (McEval), code reasoning (CRUXEval), and math reasoning (GSM8K, OlympiadBench). It also maintains strong general abilities on MMLU, CEval, and other benchmarks. The models are released under Apache 2.0 license. Next steps include a 32B version and exploration of code-centric reasoning models.