---
title: "Qwen 3 GSPO & DeepSeek GRPO — LLM Reasoning"
description: "See how DeepSeek's GRPO turns a group's rewards into advantages, and how Qwen's GSPO swaps token ratios for one length-normalized sequence ratio."
canonical_url: "https://fanout.sh/ai/lessons/reinforcement-learning-qwen-deepseek-grpo"
md_url: "https://fanout.sh/ai/lessons/reinforcement-learning-qwen-deepseek-grpo.md"
access: "public"
---

# Qwen 3 GSPO & DeepSeek GRPO — LLM Reasoning

See how DeepSeek's GRPO turns a group's rewards into advantages, and how Qwen's GSPO swaps token ratios for one length-normalized sequence ratio.

## Public overview

Ask a model the same arithmetic question four times. A checker gives reward 1 for a correct final answer and 0 otherwise. Suppose the rewards are [1, 0, 1, 0].

The model produced both successful and unsuccessful attempts. We can compare them within this group to form a learning signal, instead of training a separate model to predict the expected reward.

This is the starting idea of Group Relative Policy Optimization , or GRPO, introduced in DeepSeekMath. We will compare its outcome-reward version with Group Sequence Policy Optimization , or GSPO, from the Qwen team. Both adjust a language model using sampled responses and rewards; they weight those responses differently.

First, compare each reward with its group

Our four rewards have mean 0.5. Each reward lies 0.5 away from that mean. Square those distances, average them, and take the square root: √0.25 = 0.5. That is their population standard deviation , a measure of spread. Subtract the mean and divide by that spread:

Rᵢ is response i's reward. The resulting advantages are [1, −1, 1, −1]. Positive means better than this group's average, not “proved good in every sense.” For outcome supervision, each generated token in one response receives that response's advantage.

What happens if all four responses earn 0? What if all four earn 1?

A final-answer checker also cannot establish that every intermediate reasoning step was correct. A response can reach the correct answer through flawed reasoning; the reward only measures what the checker actually checks.

The difference is where we form the ratio

Save the probability assigned to each generated token by the old model that sampled it. After an update, recompute the probability of that same token at the same prefix under the current model.

For one two-token response, suppose those probabilities are:

Token position Old probability Current probability Current / old --- ---: ---: ---: 1 0.20 0.40 2.0 2 0.50 0.25 0.5

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md