A Survey of Reinforcement Learning for Large Reasoning Models
We welcome everyone to open an issue for any related work we haven’t discussed, and we’ll try to address it in the next release!
🎉 News
- [2026-07-31] 🎉 First OpenRSI release: Frontis-MA1 (35B / 30B, with GGUF derivatives), the OpenMLE stack (Gym / RL / Evo), and the OpenMLE Tasks and OpenMLE SFT Traces datasets. Check it out: GitHub.
- [2026-06-25] 🎉 Our survey Self-Improving Agents in the Era of Experience: A Survey of Self- to Meta-Evolution is now available on OpenReview. Check it out: GitHub and OpenReview.
- [2025-11-05] 🔥 Excited to release our paper list about Memory for Agents, covering breakthroughs in Context Management and Learning from Experience powering self-improving AI agents. Check it out: GitHub
- [2025-10] 🎉 Honored to give talks at BAAI, Qingke Talk and Tencent Wiztalk! Here are the slides.
- [2025-09-18] 🎉 We update the full list of papers in the category structure of the survey!
- [2025-09-12] 🎉 Our survey was ranked #1 Paper of the Day on 🤗 Hugging Face Daily Papers!
- [2025-09-11] 🔥 Excited to release our RL for LRMs Survey! We’ll be updating the full list of papers in with a new category structure soon. Check it out: Paper.
- [2025-08-15] 🔥 Introducing SSRL: an investigation for Agentic Search RL without reliance on external search engine. Check it out: GitHub and Paper.
- [2025-05-27] 🔥 Introducing MARTI: A Framework for LLM-based Multi-Agent Reinforced Training and Inference. Check it out: Github.
- [2025-04-23] 🔥 Introducing TTRL: an open-source solution for online RL on data without ground-truth labels, especially test data. Check it out: Github and Paper.
- [2025-03-20] 🔥 We are excited to introduce collection of papers and projects on RL for reasoning models!
🎈 Citation
If you find this survey helpful, please cite our work:
@article{zhang2025survey,
title={A survey of reinforcement learning for large reasoning models},
author={Zhang, Kaiyan and Zuo, Yuxin and He, Bingxiang and Sun, Youbang and Liu, Runze and Jiang, Che and Fan, Yuchen and Tian, Kai and Jia, Guoli and Li, Pengfei and others},
journal={arXiv preprint arXiv:2509.08827},
year={2025}
}
📖 Contents
- A Survey of Reinforcement Learning for Large Reasoning Models
- 🎉 News
- 🎈 Citation
- 📖 Contents
- 🗺️ Overview
- 📄 Paper List
- Frontier Models
- Reward Design
- Policy Optimization
- Sampling Strategy
- Training Resource
- Static Corpus (Code)
- Static Corpus (STEM)
- Static Corpus (Math)
- Static Corpus (Agent)
- Static Corpus (Mix)
- Dynamic Environment (Rule-based)
- Dynamic Environment (Code-based)
- Dynamic Environment (Game-based)
- Dynamic Environment (Model-based)
- Dynamic Environment (Ensemble-based)
- RL Infrastructure (Primary)
- RL Infrastructure (Secondary)
- Applications
- 🌟 Acknowledgment
- ✨ Star History
🗺️ Overview
Our survey provides a comprehensive examination of Reinforcement Learning for Large Reasoning Models.
We organize the survey into five main sections:
- Foundational Components: Reward design, policy optimization, and sampling strategies
- Foundational Problems: Key debates and challenges in RL for LRMs
- Training Resources: Static corpora, dynamic environments, and infrastructure
- Applications: Real-world implementations across diverse domains
- Future Directions: Emerging research opportunities and challenges
📄 Paper List
Frontier Models
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-08 | Intern-S1 |
Intern-S1: A Scientific Multimodal Foundation Model | ||
| 2025-08 | GLM-4.5 |
GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models | ||
| 2025-08 | gpt-oss |
gpt-oss-120b & gpt-oss-20b Model Card | ||
| 2025-08 | InternVL3.5 |
InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency | ||
| 2025-07 | Kimi K2 |
Kimi K2: Open Agentic Intelligence | ||
| 2025-07 | Step 3 |
Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding | ||
| 2025-07 | GLM-4.1V-Thinking |
GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning | ||
| 2025-07 | Skywork-R1V3 |
Skywork-R1V3 Technical Report | ||
| 2025-07 | GLM-4.5V |
GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning | ||
| 2025-06 | Magistral |
Magistral | - | |
| 2025-06 | Minimax-M1 |
MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention | ||
| 2025-05 | MiMo |
MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining | ||
| 2025-05 | Qwen3 |
Qwen3 Technical Report | ||
| 2025-05 | Llama-Nemotron-Ultra |
Llama-Nemotron: Efficient Reasoning Models | ||
| 2025-05 | INTELLECT-2 |
INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning | - | |
| 2025-05 | Hunyuan-TurboS |
Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought | ||
| 2025-05 | Skywork OR-1 |
Skywork Open Reasoner 1 Technical Report | ||
| 2025-04 | Phi-4 Reasoning |
Phi-4-reasoning Technical Report | - | |
| 2025-04 | Skywork-R1V2 |
Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoning | ||
| 2025-04 | InternVL3 |
InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models | ||
| 2025-03 | ORZ |
Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model | ||
| 2025-01 | DeepSeek-R1 |
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning | ||
| - | QwQ |
QwQ-32B: Embracing the Power of Reinforcement Learning | ||
| - | Seed-OSS |
Seed-OSS Open-Source Models | ||
| - | ERNIE-4.5-Thinking |
ERNIE 4.5 Technical Report | - |
Reward Design
Generative Rewards
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2026-06 | - | Steer, Don't Solve: Training Small Critic Models for Large Code Agents | ||
| 2025-08 | CAPO |
CAPO: Towards Enhancing LLM Reasoning through Verifiable Generative Credit Assignment | ||
| 2025-08 | CompassVerifier |
CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward | ||
| 2025-08 | Cooper |
Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models | ||
| 2025-08 | ReviewRL |
ReviewRL: Towards Automated Scientific Review with RL | ||
| 2025-08 | Rubicon |
Reinforcement Learning with Rubric Anchors | - | |
| 2025-08 | RuscaRL |
Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning | - | |
| 2025-07 | OMNI-THINKER |
OMNI-THINKER: Scaling Cross-Domain Generalization in LLMs via Multi-Task RL with Hybrid Rewards | - | |
| 2025-07 | URPO |
URPO: A Unified Reward & Policy Optimization Framework for Large Language Models | - | |
| 2025-07 | RaR |
Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains | - | |
| 2025-07 | RLCF |
Checklists Are Better Than Reward Models For Aligning Language Models | - | |
| 2025-07 | PCL |
Post-Completion Learning for Language Models | - | |
| 2025-07 | K2 |
KIMI K2: OPEN AGENTIC INTELLIGENCE | - | |
| 2025-07 | LIBRA |
LIBRA: ASSESSING AND IMPROVING REWARD MODEL BY LEARNING TO THINK | - | |
| 2025-07 | TP-GRPO |
Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learner | ||
| 2025-06 | RewardAnything |
RewardAnything: Generalizable Principle-Following Reward Models | ||
| 2025-06 | Writing-Zero |
Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards | - | |
| 2025-06 | Critique-GRPO |
Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback | ||
| 2025-06 | PAG |
PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier | - | |
| 2025-06 | GRAM |
GRAM: A Generative Foundation Reward Model for Reward Generalization | ||
| 2025-06 | ProxyReward |
From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation | - | |
| 2025-06 | QA-LIGN |
QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA | - | |
| 2025-05 | RM-R1 |
RM-R1: Reward Modeling as Reasoning | ||
| 2025-05 | J1 |
J1: Incentivizing Thinking in LLM-as-a-Judge via RL | - | |
| 2025-05 | TinyV |
TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning | ||
| 2025-05 | General-Reasoner |
General-reasoner: Advancing llm reasoning across all domains | - | |
| 2025-05 | RRM |
Reward Reasoning Model | - | |
| 2025-05 | RL Tango |
RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning | ||
| 2025-05 | Think-RM |
Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models | ||
| 2025-04 | JudgeLRM |
JudgeLRM: Large Reasoning Models as a Judge | ||
| 2025-04 | GenPRM |
GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning | ||
| 2025-04 | DeepSeek-GRM |
Inference-Time Scaling for Generalist Reward Modeling | - | |
| 2025-04 | AIR |
AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Dataset | - | |
| 2025-04 | Pairwise-RL |
A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization | - | |
| 2025-04 | xVerify |
xVerify: Efficient Answer Verifier for Reasoning Model Evaluations | ||
| 2025-04 | Seed-Thinking-v1.5 |
Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning | - | |
| 2025-04 | ThinkPRM |
Process Reward Models That Think | ||
| 2025-03 | - | Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains | - | |
| 2025-02 | - | Self-rewarding correction for mathematical reasoning | ||
| 2024-10 | GenRM |
Generative Reward Models | - | |
| 2024-08 | CLoud |
Critique-out-Loud Reward Models | ||
| 2024-08 | Generative Verifier |
Generative Verifiers: Reward Modeling as Next-Token Prediction | - | |
| 2024-01 | Self-Rewarding LM |
Self-Rewarding Language Models | - | |
| 2023-10 | Auto-J |
Generative Judge for Evaluating Alignment | ||
| 2023-06 | Judge LLM-as-a-Judge |
Judging llm-as-a-judge with mt-bench and chatbot arena |
Dense Rewards
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2026-09 | DRACO |
DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training | ||
| 2025-09 | Tree-GRPO |
Tree Search for LLM Agent Reinforcement Learning | ||
| 2025-09 | AttnRL |
Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models | ||
| 2025-09 | TARL |
Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agents | - | |
| 2025-09 | PROF |
Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training | ||
| 2025-09 | HICRA |
Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning | - | |
| 2025-08 | KlearReasoner |
Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization | ||
| 2025-08 | CAPO |
CAPO: Towards Enhancing LLM Reasoning through Verifiable Generative Credit Assignment | ||
| 2025-08 | GTPO & GRPO-S |
GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy | - | |
| 2025-08 | VSRM |
Promoting Efficient Reasoning with Verifiable Stepwise Reward | - | |
| 2025-08 | G-RA |
Stabilizing Long-term Multi-turn Reinforcement Learning with Gated Rewards | - | |
| 2025-08 | SSPO |
SSPO: Self-traced Step-wise Preference Optimization for Process Supervision and Reasoning Compression | - | |
| 2025-08 | AIRL-S |
Your Reward Function for RL is Your Best PRM for Search: Unifying RL and Search-Based TTS | - | |
| 2025-08 | TreePO |
TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling | ||
| 2025-08 | MUA-RL |
MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use | - | |
| 2025-07 | SPRO |
Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement | - | |
| 2025-07 | FR3E |
First Return, Entropy-Eliciting Explore | - | |
| 2025-07 | ARPO |
Agentic Reinforced Policy Optimization | ||
| 2025-07 | TP-GRPO |
Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learner | ||
| 2025-06 | TreeRPO |
TreeRPO: Tree Relative Policy Optimization | ||
| 2025-06 | TreeRL |
TreeRL: LLM Reinforcement Learning with On-Policy Tree Search | ||
| 2025-06 | Entropy Advantage |
Reasoning with Exploration: An Entropy Perspective on Reinforcement Learning for LLMs | - | |
| 2025-06 | ReasonFlux-PRM |
ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs | ||
| 2025-05 | S-GRPO |
S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models | - | |
| 2025-05 | GiGPO |
Group-in-Group Policy Optimization for LLM Agent Training | ||
| 2025-05 | - | Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit Assignment | - | |
| 2025-05 | Tango |
RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning | ||
| 2025-05 | StepSearch |
StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization | ||
| 2025-05 | - | Aligning Dialogue Agents with Global Feedback via Large Language Model Reward Decomposition | - | |
| 2025-05 | Tool-Star |
Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning | ||
| 2025-05 | SPA-RL |
SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution | ||
| 2025-05 | SPO |
Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Mode | ||
| 2025-04 | GenPRM |
GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning | ||
| 2025-04 | PURE |
Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning | ||
| 2025-03 | MRT |
Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning | ||
| 2025-03 | SWEET-RL |
SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks | ||
| 2025-02 | PRIME |
Process Reinforcement through Implicit Rewards | ||
| 2024-12 | Implicit PRM |
Free Process Rewards without Process Labels | ||
| 2024-10 | VinePPO |
VinePPO: Refining Credit Assignment in RL Training of LLMs | ||
| 2024-10 | PAV |
Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning | - | |
| 2024-04 | - | From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function | - | |
| 2024-03 | GELI |
Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedback | - | |
| 2023-12 | Math-Shepherd |
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations | - | |
| 2023-05 | PRM800K |
Let's Verify Step by Step | ||
| 2022-11 | - | Solving math word problems with process- and outcome-based feedback | - |
Unsupervised Rewards
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-09 | Vision-Zero |
Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play | ||
| 2025-08 | Co-Reward |
Co-Reward: Self-supervised Reinforcement Learning for Large Language Model Reasoning via Contrastive Agreement | ||
| 2025-08 | SQLM |
Self-Questioning Language Models | ||
| 2025-08 | R-zero |
R-Zero: Self-Evolving Reasoning LLM from Zero Data | ||
| 2025-08 | ETTRL |
ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanism | - | |
| 2025-07 | RLSF |
Post-Training Large Language Models via Reinforcement Learning from Self-Feedback | - | |
| 2025-06 | RLSC |
Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models | - | |
| 2025-06 | RPT |
Reinforcement Pre-Training | - | |
| 2025-06 | CoVo |
Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning | ||
| 2025-06 | SEAL |
Self-Adapting Language Models | - | |
| 2025-06 | Spurious Rewards |
Spurious Rewards: Rethinking Training Signals in RLVR | ||
| 2025-06 | No Free Lunch |
No Free Lunch: Rethinking Internal Feedback for LLM Reasoning | - | |
| 2025-05 | Absolute Zero |
Absolute Zero: Reinforced Self-play Reasoning with Zero Data | ||
| 2025-05 | EM-RL |
The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning | ||
| 2025-05 | SSR-Zero |
SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translation | ||
| 2025-05 | - | Surrogate Signals from Format and Length: Reinforcement Learning for Solving Mathematical Problems without Ground Truth Answers | ||
| 2025-05 | RLIF |
Learning to Reason without External Rewards | ||
| 2025-05 | SeRL |
SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data | ||
| 2025-05 | SRT |
Can Large Reasoning Models Self-Train? | ||
| 2025-05 | RENT-RL |
Maximizing Confidence Alone Improves Reasoning | ||
| 2025-04 | EMPO |
Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization | ||
| 2025-04 | TRANS-ZERO |
TRANS-ZERO: Self-Play Incentivizes Large Language Models for Multilingual Translation Without Parallel Data | ||
| 2025-04 | TTRL |
TTRL: Test-Time Reinforcement Learning | ||
| 2025-04 | One-Shot-RLVR |
Reinforcement Learning for Reasoning in Large Language Models with One Training Example | ||
| 2025-02 | CAGSR |
A Self-Supervised Reinforcement Learning Approach for Fine-Tuning Large Language Models Using Cross-Attention Signals | - | |
| 2024-07 | MINIMO |
Learning Formal Mathematics From Intrinsic Motivation |
Rewards Shaping
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-09 | CDE |
CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models | - | |
| 2025-09 | DARLING |
Jointly Reinforcing Diversity and Quality in Language Model Generations | ||
| 2025-09 | DRER |
Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL | - | |
| 2025-09 | OBE |
Outcome-based Exploration for LLM Reasoning | - | |
| 2025-08 | Pass@kTraining |
Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models | ||
| 2025-05 | PKPO |
Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems | - | |
| 2025-05 | rl-without-gt |
Surrogate Signals from Format and Length: Reinforcement Learning for Solving Mathematical Problems without Ground Truth Answers | ||
| 2025-03 | CrossDomain-RLVR |
Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains | - | |
| 2025-01 | DeepSeek-R1 |
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning | ||
| 2024-09 | Qwen2.5-Math |
Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement |
Policy Optimization
Policy Gradient Objective
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2017-07 | PPO |
Proximal policy optimization algorithms | - | |
| - | PG |
Policy gradient methods for reinforcement learning with function approximation. | - | |
| - | REINFORCE |
Simple statistical gradient-following algorithms for connectionist reinforcement learning | - | |
| - | TRPO |
Trust region policy optimization | - |
Critic-based Algorithms
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-08 | VL-DAC |
Enhancing Vision-Language Model Training with Reinforcement Learning in Synthetic Worlds for Real-World Success | ||
| 2025-08 | VRPO |
VRPO:Rethinking Value Modeling for Robust RL Training under Noisy Supervision | - | |
| 2025-05 | VerIPO |
VerIPO: Long Reasoning Video-R1 Model with Iterative Policy Optimization | ||
| 2025-04 | VAPO |
Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks | - | |
| 2025-03 | VCPPO |
What’s Behind PPO’s Collapse in Long-CoT? Value Optimization Holds the Secret | - | |
| 2025-03 | Open reasoner-zero |
open reasoner-zero: An open source approach to scaling up reinforcement learning on the base model | ||
| 2025-02 | PRIME |
PROCESS REINFORCEMENT THROUGH IMPLICIT REWARDS | ||
| 2024-12 | Implicit PRM |
FREE PROCESS REWARDS WITHOUT PROCESS LABELS | ||
| 2023-12 | Math-shepherd |
Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations | - | |
| 2015-06 | GAE |
High-dimensional continuous control using generalized advantage estimation | - | |
| - | Autopsv |
Autopsv: Automated process-supervised verifier. |
Critic-Free Algorithms
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-09 | UPGE |
Towards a Unified View o fLarge Language Model Post-Training | ||
| 2025-09 | SPO |
Single-stream Policy Optimization | - | |
| 2025-08 | LitePPO |
Part I: Tricks or Traps? A Deep Dive into RLfor LLM Reasoning | - | |
| 2025-07 | R1-RE |
R1-RE: Cross-Domain Relation Extraction with RLVR | - | |
| 2025-07 | GSPO |
Group Sequence Policy Optimization | - | |
| 2025-06 | CISPO |
MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention | ||
| 2025-05 | KRPO |
Kalman Filter Enhanced Group Relative Policy Optimization for Language Model Reasoning | ||
| 2025-05 | CPGD |
CPGD:Toward Stable Rule-based Reinforcement Learning for Language Models | ||
| 2025-05 | NFT |
Bridging Supervised Learning and Reinforcement Learning in Math Reasoning | - | |
| 2025-05 | Clip-Cov/KL-Cov |
The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models | ||
| 2025-03 | OpenVLThinker |
OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles | ||
| 2025-03 | DAPO |
DAPO: an Open-Source LLM Reinforcement Learning System at Scale | ||
| 2025-03 | Dr. GRPO |
Understanding R1-Zero-Like Training: A Critical Perspectiv | ||
| 2025-01 | Kimi k1.5 |
Kimi k1.5: Scaling Reinforcement Learning with LLMs | - | |
| 2024-02 | RLOO |
Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms | - | |
| 2024-02 | GRPO |
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models | ||
| 2023-10 | ReMax |
ReMax: A Simple, Effective, and Efficient Method for Aligning Large Language Models | ||
| - | REINFORCE |
Simple statistical gradient-following algorithms for connectionist reinforcement learning | - | |
| - | REINFORCE++ |
REINFORCE++: An Efficient RLHF Algorithm with Robustnessto Both Prompt and Reward Models | ||
| - | VinePPO |
VINEPPO: UNLOCKING RL POTENTIAL FOR LLM REASONING THROUGH REFINED CREDIT ASSIGNMENT | ||
| - | FlashRL |
Fast RL training with Quantized Rollouts |
Off-policy Optimization
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-09 | BRIDGE |
Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning | ||
| 2025-09 | HPT |
Towards a Unified View of Large Language Model Post-Training | ||
| 2025-08 | DFT |
On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification | ||
| 2025-08 | RED |
Recall-Extend Dynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integration | ||
| 2025-07 | Prefix‑RFT |
Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling | - | |
| 2025-07 | ReMix |
Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model | ||
| 2025-06 | ReLIFT |
Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions | ||
| 2025-06 | BREAD |
BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning | - | |
| 2025-06 | SRFT |
SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning | - | |
| 2025-05 | AMPO |
Adaptive Thinking via Mode Policy Optimization for Social Language Agents | ||
| 2025-05 | UFT |
UFT: Unifying Supervised and Reinforcement Fine-Tuning | ||
| 2025-04 | LUFFY |
Learning to Reason under Off-Policy Guidance | ||
| 2025-03 | SPO |
Soft Policy Optimization: Online Off-Policy RL for Sequence Models | ||
| 2025-03 | TOPR |
TAPERED OFF-POLICY REINFORCE Stable and efficient reinforcement learning for LLMs | - | |
| 2024-05 | IFT |
Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process | ||
| 2023-05 | DPO |
Direct Preference Optimization: Your Language Model is Secretly a Reward Model | - | |
| 2015-11 | - | Fixed point quantization of deep convolutional networks | - | |
| - | - | Your Efficient RL Framework Secretly Brings You Off-Policy RL Training |
Off-policy Optimization (Exp replay)
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-09 | SAPO |
Sharing is Caring: Efficient LM Post-Training with Collective RL Experience Sharing | - | |
| 2025-09 | SEELE |
Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding | ||
| 2025-08 | Memory-R1 |
Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning | - | |
| 2025-07 | RLEP |
RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning | ||
| 2025-06 | EFRame |
EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework | ||
| 2025-05 | ARPO |
ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay | ||
| 2025-04 | - | Improving RL Exploration for LLM Reasoning through Retrospective Replay | - |
Regularization Objectives
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-10 | ASPO |
ASPO: Asymmetric Importance Sampling Policy Optimization | ||
| 2025-09 | CE-GPPO |
CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning | ||
| 2025-09 | CDE |
CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models | - | |
| 2025-09 | DPH RL |
The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward | ||
| 2025-09 | empgseed-seed |
Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents | - | |
| 2025-07 | Archer |
Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR | ||
| 2025-06 | Bingo |
Bingo: Boosting Efficient Reasoning of LLMs via Dynamic and Significance-based Reinforcement Learning | ||
| 2025-06 | HighEntropy RL |
Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning | - | |
| 2025-06 | Entropy RL |
Reasoning with Exploration: An Entropy Perspective on Reinforcement Learning for LLMs | - | |
| 2025-06 | ALP RL |
Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning | - | |
| 2025-05 | DisCO |
DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization | ||
| 2025-05 | Skywork OR1 |
Skywork Open Reasoner 1 Technical Report | ||
| 2025-05 | Entropy Mechanism |
The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models | ||
| 2025-05 | ProRL |
ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models | - | |
| 2025-05 | Short RL |
Efficient RL Training for Reasoning Models via Length-Aware Optimization | ||
| 2025-03 | DAPO |
DAPO: An Open-Source LLM Reinforcement Learning System at Scale | - | |
| 2025-03 | L1 |
L1: Controlling how long a reasoning model thinks with reinforcement learning |
Sampling Strategy
Dynamic and Structured Sampling
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-10 | EEPO |
EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget | ||
| 2025-09 | AttnRL |
Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models | - | |
| 2025-09 | DACE |
Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learning | - | |
| 2025-09 | Parallel-R1 |
Parallel-R1: Towards Parallel Thinking via Reinforcement Learning | ||
| 2025-08 | G^2RPO-A |
G^2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidanc | ||
| 2025-08 | RuscaRL |
Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning | - | |
| 2025-08 | TreePO |
TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling | ||
| 2025-07 | ARPO |
Agentic Reinforced Policy Optimization | ||
| 2025-06 | TreeRPO |
TreeRPO: Tree Relative Policy Optimization | ||
| 2025-06 | E2H |
Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning | - | |
| 2025-06 | TreeRL |
TreeRL: LLM Reinforcement Learning with On-Policy Tree Search | ||
| 2025-05 | ToTRL |
ToTRL: Unlock LLM Tree-of-Thoughts Reasoning Potential through Puzzles Solving | - | |
| 2025-03 | DARS |
DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversal | ||
| 2025-03 | DAPO |
DAPO: An Open-Source LLM Reinforcement Learning System at Scale | ||
| 2025-02 | PRIME |
Process Reinforcement through Implicit Rewards | ||
| - | POLARIS |
POLARIS: A POst-training recipe for scaling reinforcement Learning on Advanced ReasonIng modelS |
Sampling Hyper-Parameters
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-08 | GFPO |
Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning | - | |
| 2025-06 | AceReason-Nemotron 1.1 |
AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy | - | |
| 2025-06 | T-PPO |
Truncated Proximal Policy Optimization | - | |
| 2025-06 | Confucius3-Math |
Confucius3-Math: A Lightweight High-Performance Reasoning LLM for Chinese K-12 Mathematics Learning | ||
| 2025-05 | E3-RL4LLMs |
Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs | ||
| 2025-05 | AceReason-Nemotron |
AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning | - | |
| 2025-05 | Pro-RL |
ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models | - | |
| 2025-03 | - | Output Length Effect on DeepSeek-R1's Safety in Forced Thinking | - | |
| 2025-03 | DAPO |
DAPO: An Open-Source LLM Reinforcement Learning System at Scale | ||
| 2025-02 | PRIME |
Process Reinforcement through Implicit Rewards | ||
| 2025-02 | - | Training Language Models to Reason Efficiently | - | |
| - | DeepScaleR |
DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL | ||
| - | POLARIS |
POLARIS: A POst-training recipe for scaling reinforcement Learning on Advanced ReasonIng modelS |
Training Resource
Static Corpus (Code)
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-05 | rStar-Coder |
rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset | ||
| 2025-04 | Z1 |
Z1: Efficient Test-time Scaling with Code | ||
| 2025-04 | OpenCodeReasoning |
OpenCodeReasoning: Advancing Data Distillation for Competitive Coding | - | |
| 2025-04 | LeetCodeDataset |
LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs | ||
| 2025-03 | KodCode |
KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding | - | |
| 2025-01 | SWE-Fixer |
SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution | ||
| 2024-12 | SWE-Gym |
Training Software Engineering Agents and Verifiers with SWE-Gym | ||
| - | Code-R1 |
Code-R1: Reproducing R1 for Code with Reliable Rewards | ||
| - | codeforces-cots |
CodeForces CoTs | - | |
| - | DeepCoder |
DeepCoder: A Fully Open-Source 14B Coder at O3-mini Level |
Static Corpus (STEM)
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-09 | SSMR-Bench |
Synthesizing Sheet Music Problems for Evaluation and Reinforcement Learning | ||
| 2025-09 | Loong |
Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifiers | ||
| 2025-07 | MegaScience |
MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning | - | |
| 2025-06 | ReasonMed |
ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning | ||
| 2025-05 | ChemCoTDataset |
Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations | - | |
| 2025-02 | NaturalReasoning |
NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions | - | |
| 2025-01 | SCP-116K |
SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domain | - |
Static Corpus (Math)
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-07 | MiroMind-M1-RL-62K |
MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization | ||
| 2025-04 | DeepMath |
DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning | ||
| 2025-04 | OpenMathReasoning |
AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset | ||
| 2025-03 | STILL-3-RL |
An Empirical Study on Eliciting and Improving R1-like Reasoning Models | ||
| 2025-03 | Light-R1 |
Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond | - | |
| 2025-03 | DAPO |
DAPO: An Open-Source LLM Reinforcement Learning System at Scale | ||
| 2025-03 | OpenReasoningZero |
Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model | ||
| 2025-02 | PRIME |
Process Reinforcement through Implicit Rewards | ||
| 2025-02 | LIMO |
Limo: Less is more for reasoning | ||
| 2025-02 | LIMR |
Limr: Less is more for rl scaling | ||
| 2025-02 | Big-MATH |
Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models | - | |
| - | NuminaMath 1.5 |
Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions | ||
| - | OpenR1-Math |
Open R1: A fully open reproduction of DeepSeek-R1 | ||
| - | DeepScaleR |
DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL | - |
Static Corpus (Agent)
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-08 | ASearcher |
Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL | - | |
| 2025-07 | WebShaper |
WebShaper: Agentically Data Synthesizing via Information-Seeking Formalization | - | |
| 2025-05 | ZeroSearch |
ZeroSearch: Incentivize the Search Capability of LLMs without Searching | ||
| 2025-04 | ToolRL |
ToolRL: Reward is All Tool Learning Needs | ||
| 2025-03 | Search-R1 |
Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning | ||
| 2025-03 | ToRL |
ToRL: Scaling Tool-Integrated RL | ||
| - | MicroThinker |
MiroVerse V0.1: A Reproducible, Full-Trajectory, Ever-Growing Deep Research Dataset | - | |
| 2025-03 | DeepRetrieval |
DeepRetrieval: Hacking Real Search Engines and Retrievers with Large Language Models via Reinforcement Learning |
Static Corpus (Mix)
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-08 | Graph-R1 |
Graph-R1: Unleashing LLM Reasoning with NP-Hard Graph Problem | - | |
| 2025-06 | RewardAnything |
RewardAnything: Generalizable Principle-Following Reward Models | ||
| 2025-06 | guru-RL-92k |
Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective | - | |
| 2025-05 | Llama-Nemotron-PT |
Llama-Nemotron: Efficient Reasoning Models | - | |
| 2025-05 | SkyWork OR1 |
Skywork Open Reasoner 1 Technical Report | ||
| 2025-03 | OpenVLThinker |
OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles | ||
| - | AM-DS-R1-0528-Distilled |
AM-DeepSeek-R1-0528-Distilled | ||
| - | dolphin-r1 |
Dolphin R1 Dataset | - | |
| - | SYNTHETIC-1/2 |
SYNTHETIC-1 Release: Two Million Collaboratively Generated Reasoning Traces from Deepseek-R1 | - |
Dynamic Environment (Rule-based)
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-06 | ProtoReasoning |
ProtoReasoning: Prototypes as the Foundation for Generalizable Reasoning in LLMs | - | |
| 2025-05 | SynLogic |
SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond | ||
| 2025-05 | Reasoning Gym |
REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards | ||
| 2025-05 | Enigmata |
Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles | ||
| 2025-02 | AutoLogi |
AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models | ||
| 2025-02 | Logic-RL |
Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning |
Dynamic Environment (Code-based)
| Date | Name | Title | Paper | Github |
|---|---|---|---|---|
| 2025-06 | AgentCPM-GUI |
AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning | ||
| 2025-06 | MedAgentGym |
MedAgentGym: Training LLM Agents for Code-Based Medical Reasoning at Scale | ||
| 2025-05 | MLE-Dojo |
MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering | ||
| 2025-05 | SWE-rebench |
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents | - | |
| 2025-05 | ZeroGUI |
ZeroGUI: Automating Online GUI Learning at Zero Human Cost | ||
| 2025-04 | R2E-Gym |
R2E-Gym: Procedural Environment Generation and Hybrid Verifiers for Scaling Open-Weights SWE Agents | ||
| 2025-03 | ReSearch |
ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning | ||
| 2025-02 | MLGym |
MLGym: A New Framework and Benchmark for Advancing AI Research Agents | ||
| 2024-07 | AppWorld |
AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents | [![GitHub Stars](https://img.shields.io/githu |