Kernelized advantage estimation improves LLM reasoning training via nonparametric value estimation

Researchers propose kernelized advantage estimation as a drop-in replacement for value network-based methods like PPO and group-based methods like GRPO in LLM RL training. The approach reduces computational overhead while maintaining low-variance policy gradient estimates.