[skyrl-train] Add GDPO Support to PPO Utils

[devpatelio]

GDPO is an extension of GRPO for multi-reward settings where we do group-wise normalization of each reward function prior to computing the advantage. This is then followed by a batch-norm across all prompts belonging to a given batch and it's respective advantages ( GDPO Paper)

Points of clarification:

• How does skyrl handle multiple reward functions (if at all)? -> we assume token-level rewards also contain an extra dimension for N objectives
• Should we port from VERL? It seems like it only supports max 2 rewards -> https://github.com/NVlabs/GDPO/blob/main/verl-GDPO/verl/trainer/ppo/ray_trainer.py

TODOs:

• add multiple reward functionality
• test GDPO performance against GRPO

[gemini-code-assist]

Code Review

This pull request introduces support for Group-wise Distributional Policy Optimization (GDPO) advantage estimation by adding a new advantage estimator, compute_gdpo_outcome_advantage, to the PPO utilities. A medium-severity vulnerability was identified in the new GDPO implementation, specifically a critical bug in the group-wise normalization step that could lead to division by zero. Additionally, a broken link to the reference paper in the docstring was found.