PPO, LLM Reasoning, Importance Ratio, Advantage

Work through PPO's clipped objective with numbers: the importance ratio, the sign of the advantage, the clip width ε, and how it maps onto LLM training.

A language model answered a prompt. We scored its response, and now want to make useful choices more likely. But if we train repeatedly on the same recorded response, how much should one batch be allowed to change the model?

Proximal Policy Optimization , or PPO, includes a method called PPO-Clip that removes some incentive to keep pushing a recorded action's probability in the same direction. We will inspect that mechanism with one action. The action could be a game move or a language model's next token.

Compare the old and current probabilities

Suppose the model that collected the response assigned probability 0.20 to a particular token at a particular prefix. After an update, the current model assigns 0.30 to that same token at the same prefix .

The ratio is 0.30 / 0.20 = 1.5. The token is now 50% more likely at that prefix. This importance ratio compares the current policy with the old policy that generated the data:

Here π is an action probability, θ means the current model parameters, s is the state or token prefix, and a is the recorded action. Keep the old probability fixed while updating the current model. The two policies are identical at the start of the batch, so the ratio starts at 1.

Now suppose this action's estimated future return is 4 and the baseline expectation is 2. Its advantage is +2: better than expected. An advantage of −2 would mean worse than expected. In a language model, a final response score can contribute to advantages for the preceding tokens; it does not identify a separate correct answer for every token.

Watch where the incentive stops

Without clipping, the policy objective for this sample would be ratio × advantage. At ratio 1.5 and advantage +2, that is 3. Increasing the ratio would keep increasing the objective.

For this example, choose a clipping width ε = 0.2. That gives the interval [0.8, 1.2]. Clipping leaves values inside the interval unchanged, replaces smaller ones with 0.8, and replaces larger ones with 1.2. PPO-Clip compares the ordinary product with a product using the clipped ratio, and takes the smaller one. We maximize this objective:

 is the estimated advantage, held fixed for this policy update. With +2 advantage, increasing the ratio from 1.2 to 1.5 no longer helps this term: it stays at 2.4.

What if the advantage is negative ? Predict the objective at ratio 0.5 and advantage −2 before opening the answer.