Thinking and Reasoning

The Purpose I Write This Blog

   Thinking models are crazily popualr nowadays. The first time I delved in this area was in September, 2023. Later I gradually forgetted this area, until Deepseek came to life. I want to keep to collect information about LLM reasoning (as well as post-training) and share my thoughts here.

💡 This post is mainly focused on general reasoning. For agentic reasoning, please refer to this post.

Reinforcement Learning

Blogs

• general
  • 一文读懂后训练：从 RLHF 到 MOPD
• algorithms
  • Reinforcement Learning from Human Feedback
  • 强化学习：从策略梯度到TRPO、PPO、DPO、GRPO
  • 人人都能看懂的RL-PPO理论知识
  • Reasoning LLM（三）：LLM+RL
  • RLHF 常见的思维误区

RL algorithms

• blogs

  • RL for LLM 高质量文章汇总
  • From GRPO to DAPO and GSPO: What, Why, and How
  • meta-RL
    • 【强化学习解惑】元强化学习（Meta-RL）是什么？它与普通强化学习有什么区别？

• general algorithms

  • (GAE) High-Dimensional Continuous Control Using Generalized Advantage Estimation
  • (DPO) Direct preference optimization: Your language model is secretly a reward model
    • 推导
  • From r to q∗: Your language model is secretly a q-function
    • DPO新作 Your Language Model is Secretly a Q-Function 解读，与OPENAI Q* 的联系？
  • (PPO) Proximal Policy Optimization Algorithms
  • (REINFORCE++) REINFORCE++: An Efficient RLHF Algorithm with Robustness to Both Prompt and Reward Models
  • (GRPO) DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  • VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
  • (DAPO) DAPO: An Open-Source LLM Reinforcement Learning System at Scale
  • (GSPO) Group Sequence Policy Optimization
  • Soft Adaptive Policy Optimization
  • (Cispo) MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention

• RL optimization tricks

  • advantage
    • GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
    • DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning
    • (RDPO) Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization
    • (SA-MRPO) Learn What’s Left, Not What’s Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization
  • interpret
    • ISO: An RLVR-Native Optimization Stack
  • learn from RMs
    • 介绍 Composer 2.5
    • SRPO ：Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning
    • (SDPO) Reinforcement Learning via Self-Distillation
    • (Critique-GRPO) Advancing LLM Reasoning with Natural Language and Numerical Feedback
  • training-inference matching
    • Predicting and Controlling Staleness in Fully Asynchronous RL Training
    • Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers
    • (IcePop) Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model
      • Small Leak Can Sink a Great Ship–Boost RL Training on MoE with IcePop!
      • IcePop: Stabilizing RL in MoE Models
  • others
    • (SPO1) Simple Policy Optimization
    • (SPO2) Single-stream Policy Optimization
    • (SPRO) Self-Guided Process Reward Optimization

• reward modeling

  • text
    • Long-Horizon Language Model Reinforcement Learning via Progressive Point Matching
    • The Verification Horizon: No Silver Bullet for Coding Agent Rewards
    • (PRM) Let’s verify step by step
    • (POLAR) Pre-Trained Policy Discriminators are General Reward Models
  • reward model for generative models
    • VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoning
    • (VideoReward) Improving Video Generation with Human Feedback
    • VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation
    • Black-Box Prompt Optimization: Aligning Large Language Models without Model Training

• mix-training

  • SFT-then-RL
    • SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
    • Learning to Reason under Off-Policy Guidance

• Distillation

  • On-Policy Distillation
    • blogs
      • On-Policy Distillation
      • 系统聊聊 On-Policy Distillation 的原理
      • 扒一扒大模型能力合版的门道和OPD
      • SFT, RL, and On-Policy Distillation Through a Distributional Lens
      • 近期关于Off Policy&On-Policy Distillation的一些总结
      • 近半年 On-Policy Distillation 的三大主流方向：一个方法解决两道难题
    • papers
      • Self-Supervised Visual On-Policy Distillation
      • OPRD: On-Policy Representation Distillation
      • VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation
      • Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation
  • Self-Distillation
    • papers
      • (U-OPSD) On-Policy Self-Distillation without Any Supervision
      • (SDPO) Reinforcement Learning via Self-Distillation
      • Self-Distillation Enables Continual Learning
      • (SDAR) Self-Distilled Agentic Reinforcement Learning
      • (OPSD) Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
  • Analysis
    • Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
    • Rethinking On-Policy Distillation of Large Language Models II: One Training Example

Engineering

• environment and data
  • Verifiable Environments Are LEGO Bricks: Recursive Composition for Reasoning Generalization
• infra
  • (verl) HybridFlow: A Flexible and Efficient RLHF Framework
    • doc(cn)
    • doc(en)
    • repo
  • slime
    • RL Scaling 时代，我们需要什么样的 RL 框架呢？
  • blogs
    • training
      • 知乎 Chayenne Zhao
      • 浅聊RL框架的勃勃生机、万物竞发
      • How we built our multi-agent research system
      • verl
        • [AI Infra] VeRL 框架入门&代码带读
        • 从零开始的verl框架解析
        • verl RL支持训练deepseek-v3 671B实习复盘(个人版)
        • OpenRLHF&Verl参数转换指南
        • verl小白解读
      • 一文深度全面解析大模型分布式并行策略：DP/TP/PP/CP/EP/SP
      • 深入理解 Megatron-LM（2）原理介绍
      • DeepSpeed zero1，zero2，zero3和FSDP区别详解
    • inference
      • SGLang：LLM推理引擎发展新方向
      • 图解大模型计算加速系列：FlashAttention V1，从硬件到计算逻辑

Analyses

• RL training
  • Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  • Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
• PPO vs. GRPO
  • GLM-5.2 放弃了开源社区曾广泛采用的 GRPO，这意味着什么？
• entropy
  • Reasoning LLM（五）：熵缩过程与能力边界
  • LLMxRL】熵坍缩与缓解策略
  • (clip/kl-cov) The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
  • (forking tokens) Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
• KL divergence
  • A Comedy of Estimators: On KL Regularization in RL Training of LLMs
  • RL 中的 KL 估计器选型：从数值无偏到梯度正确
• RL v.s. SFT
  • Sft memorizes, rl generalizes: A comparative study of foundation model post-training
  • RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
  • 3.2 统一视角理解从 SFT 到 RL
  • All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning
  • Generalist Reward Models: Found Inside Large Language Models
  • (DFT) On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
    • 从 SFT 到 RL：一步步看清它们的联系
  • (NFT) Bridging Supervised Learning and Reinforcement Learning in Math Reasoning
• others (training dynamics, mechanisms, …)
  • How’s it going? Reinforcement learning in language models recruits a functional welfare axis

Thinking Models and Methods

models

• DeepSeek-V3 Technical Report
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
• DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
• Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning
• Kimi k1.5: Scaling Reinforcement Learning with LLMs
• Kimi K2: Open Agentic Intelligence
• Kimi K2.5: Visual Agentic Intelligence
• KIMI K3: Open Frontier Intelligence
  • code
• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
• GLM-5: from Vibe Coding to Agentic Engineering
• GLM-5.2: Built for Long-Horizon Tasks
• MiniCPM
  • code
• MAI-Thinking-1: Building a Hill-Climbing Machine
• (Ring-1T) Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model
• Skywork Open Reasoner 1 Technical Report

text-based reasoning

• explicit reasoning
  • ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline
• implicit reasoning
  • (Coconut) Training Large Language Models to Reason in a Continuous Latent Space
  • Emergence of Superposition: Unveiling the Training Dynamics of Chain of Continuous Thought
  • Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought
  • Think before you speak: Training language models with pause tokens
  • Pause Tokens Strictly Increase the Expressivity of Constant-Depth Transformers
• overthinking
  • blog
    • Controlling Reasoning Effort in LLMs
    • 自适应快慢思考推理模型（Adaptive Reasoning Model）：Qwen3混合思考->字节AdaCoT->清华AdaptThinking
  • survey
    • Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
    • (repo) Awesome-Efficient-Reasoning-LLMs
  • papers
    • Qwen3 Technical Report
    • AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learning
    • AdaptThink: Reasoning Models Can Learn When to Think
• parallel thinking
  • Deep Think with Confidence
• others
  • From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones
• blogs
  • 自顶向下方式深度解读 DeepSeek-R1，内含大量细节
  • MLA(1)：从代码角度学习和彻底理解 DeepSeek MLA 算法
  • 从头理解思考模型（LLM based Reasoning Model），O1，DeepSeek R1，Kimi K1.5

visual reasoning

• image understanding
  • blog
    • 万字长文总结 Think with image 最新进展
    • Thinking with Images 小结
  • survey
    • Awesome-Multimodal-Large-Language-Models
    • Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers
  • papers
    • general
      • Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception
      • Thyme: Think Beyond Images
      • CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization
      • CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoning
      • Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearch
      • DeepEyesV2: Toward Agentic Multimodal Model
      • DeepEyes: Incentivizing “Thinking with Images” via Reinforcement Learning
      • PyVision-RL: Forging Open Agentic Vision Models via RL
      • Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search
      • Chain-of-Focus: Adaptive Visual Search and Zooming for Multimodal Reasoning via RL
      • GRIT: Teaching MLLMs to Think with Images
      • VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selection
      • VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use
      • Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration
      • Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models
      • VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoning
      • $V^{*}$: Guided Visual Search as a Core Mechanism in Multimodal LLMs
    • generation for comprehension
      • Thinking with Generated Images
      • Visual Planning: Let’s Think Only with Images
    • implicit reasoning
      • Latent Visual Reasoning
      • Do multimodal models imagine electric sheep?
      • DeepLatent: Think with Images via Parallel Latent Visual Reasoning
      • Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens
      • Beyond Visual Memory: Mechanistic Diagnostics of Latent Visual Reasoning
      • Cosine Misleads: Auxiliary Losses Reshape Vision Language Models, Not Their Latents
    • special
      • VisLens: Single-Pass Interpretable Visual Search for Multimodal LLMs
      • ARC Prize 2024
      • ARC Is a Vision Problem!
      • Think Visually, Reason Textually: Vision-Language Synergy in ARC
      • The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
    • applications
      • (GeoGuessr) How to turn a game into an RL environment
  • analyses
    • What Does Vision Tool-use Reinforcement Learning Really Learn? Disentangling Tool-Induced and Intrinsic Effects for Crop-and-Zoom
      • RL到底能不能教会模型Think with Images？
    • How Modality Shapes Perception and Reasoning: A Study of Error Propagation in ARC-AGI
    • Do AI Models Perform Human-like Abstract Reasoning Across Modalities?
• video understanding
  • survey
    • Video Understanding: From Geometry and Semantics to Unified Models
  • methods
    • FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding
    • (VILAMP) Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation
    • Video-Thinker: Sparking" Thinking with Videos" via Reinforcement Learning
  • streaming
    • Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously
    • StreamingVLM: Real-Time Understanding for Infinite Video Streams
    • STREAMMIND: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition

Speech LLMs

• LLM-based
  • end-to-end
    • Learning When to Think While Listening
    • Stitch: Simultaneous thinking and talking with chunked reasoning for spoken language models
    • Can Speech LLMs Think while Listening?
  • cascaded
    • Introducing Seed Full-Duplex Speech LLM: Attentive Listening, Robust Interference Suppression, Enabling More Natural Interaction
    • LTS-VoiceAgent: A Listen-Think-Speak Framework for Efficient Streaming Voice Interaction via Semantic Triggering and Incremental Reasoning
    • StreamingThinker: Large Language Models Can Think While Reading
    • SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models
• multi-modal
  • LiveServe: Interaction-Aware Serving for Real-Time Omni-Modal LLMs
• speculative decoding
  • 详细谈谈DSpark投机解码的原理
  • SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative Decoding

Others

• [蒙特卡洛搜索树] MCT Self-Refine (MCTSr)的算法（包含代码理解）
• 聊聊推理模型中的PRMs与MCTS

Analyses on reasoning

general

• Is AI Reasoning Right for the Wrong Reasons?
• SHAPE of Chain-of-Thought in Math Reasoning
• Case-Based or Rule-Based: How Do Transformers Do the Math?

interpretability

• Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction
• How Reinforcement Learning After Next-Token Prediction Facilitates Learning
• Base Models Know How to Reason, Thinking Models Learn When
• Topology of Reasoning: Understanding Large Reasoning Models through Reasoning Graph Properties
• Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning
• Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning
• Thought Anchors: Which LLM Reasoning Steps Matter?
• Understanding Reasoning in Thinking Language Models via Steering Vectors
• Chain-of-Thought Is Not Explainability
• Unveiling the Mechanisms of Explicit CoT Training: How Chain-of-Thought Enhances Reasoning Generalization
• How Do LLMs Perform Two-Hop Reasoning in Context?

theories

• Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought

safety

• Stealing Reasoning Traces from Proprietary LLM APIs