SRPO: Kuaishou's New RL Method 10x More Efficient Than GRPO for LLM Training
Researchers from Kuaishou's Kwaipilot team have introduced SRPO, a two-stage reinforcement learning framework that achieves DeepSeek-R1-Zero-level performance in math and code domains with only one-tenth of the training steps. SRPO addresses cross-domain optimization conflicts, low sample efficiency, and premature saturation through staged training and history resampling.
Kuaishou's Kwaipilot team developed SRPO (Two-Staged history-Resampling Policy Optimization), a reinforcement learning framework for LLMs that surpasses standard GRPO. SRPO employs a two-stage training paradigm: first, training on challenging math data to elicit reasoning, then integrating code data for skill combination. History Resampling filters out easy samples with zero reward variance, ensuring effective gradient updates. Experiments show that SRPO-Qwen-32B achieves scores of 50 on AIME24 and 41.6 on LiveCodeBench, comparable to DeepSeek-R1-Zero-32B but with 10x fewer training steps. The model exhibits reflective behaviors like rechecking, hesitation, and exploration, and spontaneously uses code to verify math solutions. The team open-sourced the model and published a technical report.
- Сокращения
- RL = Reinforcement Learning — обучение с подкреплением
- GRPO = Group Relative Policy Optimization — групповая относительная оптимизация политики
- LLM = Large Language Model — большая языковая модель
- SRPO = Two-Staged history-Resampling Policy Optimization — двухэтапная оптимизация политики с пересэмплированием истории
Source: Synced —
original