GSPO: scalable reinforcement learning for language models

Alibaba Qwen has announced the Group Sequence Policy Optimization (GSPO) algorithm, designed for scaling reinforcement learning (RL) of language models. The developers noted that existing RL algorithms, particularly GRPO, suffer from severe instability during long training runs and can lead to irreversible model collapse. GSPO defines the importance ratio based on sequence likelihood and performs clipping and optimization at the sequence level rather than at the individual token level. Cold-start experiments using Qwen3-30B-A3B-Base showed that GSPO achieves significantly higher training efficiency than GRPO and provides continuous performance improvement as computational cost increases. A key advantage of GSPO is the complete elimination of the need for the Routing Replay strategy, which was required for stable training of Mixture-of-Experts (MoE) models with GRPO. Furthermore, sequence-level optimization makes GSPO more tolerant to precision errors, simplifying infrastructure and allowing direct use of likelihood from inference engines without recalculation. GSPO has been successfully applied in large-scale RL training of Qwen3 models (Instruct, Coder, Thinking), leading to substantial performance improvements in benchmarks such as AIME'24, LiveCodeBench, and CodeForces. The authors emphasize that the proportion of clipped tokens in GSPO is two orders of magnitude higher than in GRPO, while training efficiency remains superior, indicating greater reliability of the sequence-level approach.