Working through group relative policy optimization. It is a policy gradient method in the PPO family, and the interesting part is what it takes out.
PPO answers good compared to what with a learned value model. A second network, about the size of the first, trained alongside it to guess expected return. Expensive to hold in memory, and one more thing that can be quietly wrong.
GRPO drops the critic. For each prompt it samples a group of completions, scores them, and compares each one against its own group. Better than its peers gets reinforced. Worse gets pushed down. The baseline comes free from samples you already drew.
The question I keep coming back to is what that signal does when every completion in a group is wrong.