SRPO: New RL Method from Kuaishou 10x More Efficient Than GRPO for LLM Training
Researchers from Kuaishou introduced SRPO, a two-stage reinforcement learning method that achieves DeepSeek-R1-Zero level performance on math and code tasks in 1/10 of the training steps. The method addresses cross-domain conflicts and inefficient sample utilization issues inherent in standard GRPO.
The Kwaipilot team from Kuaishou has released a new reinforcement learning architecture—Two-Staged history-Resampling Policy Optimization (SRPO)—that addresses key issues of standard GRPO: conflicts between math and coding tasks, low efficiency due to uniform rewards within a group, and premature saturation. SRPO consists of two stages: in the first stage, the model is trained only on challenging math data to develop long reasoning chains; in the second stage, programming tasks are added to reinforce skills. To improve efficiency, History Resampling is used—a dataset reconstruction that excludes overly simple examples where all candidate answers are correct. As a result, the SRPO-Qwen-32B model, based on Qwen2.5-32B, achieved results on the AIME24 (50 points) and LiveCodeBench (41.6 points) benchmarks comparable to DeepSeek-R1-Zero-32B, while requiring only 1/10 of the training steps. During experiments, the model spontaneously learned to use programming code to verify mathematical solutions.
Source: Synced —
original