"PILAF: Optimal Human Preference Sampling for Reward Modeling"

Below podcast on this paper is generated with Google's Illuminate.

https://arxiv.org/abs/2502.04270

The challenge in Reinforcement Learning from Human Feedback (RLHF) is that current methods for collecting preference data for reward modeling are inefficient and do not directly optimize for true human values. This paper addresses the misalignment between reward model training and maximizing the actual human preference objective in RLHF.

This paper proposes Policy-Interpolated Learning for Aligned Feedback (PILAF). PILAF is a novel sampling strategy for preference labeling. It aligns preference learning with the goal of maximizing the underlying true human reward.

-----

📌 PILAF directly tackles the objective misalignment in Reinforcement Learning from Human Feedback. It aligns reward model training with the true preference objective by shaping the preference sampling, enhancing learning efficiency.

📌 Statistically, PILAF strategically samples data in directions of maximal objective sensitivity. This reduces variance in preference data, leading to more robust and data-efficient reward model training.

📌 PILAF's practical strength lies in its simple implementation and lack of hyperparameters. This allows for immediate improvement in existing Direct Preference Optimization pipelines with minimal overhead.

----------

Methods Explored in this Paper 🔧:

→ Introduces Theoretically Grounded Policy-Interpolated Learning for Aligned Feedback (T-PILAF). T-PILAF generates response pairs by interpolating between the current policy and a reference policy. This balances exploration and exploitation during data collection.

→ T-PILAF uses two modified policies, π+ and π−, derived from the current policy π. π+ encourages exploration in areas favored by π, and π− explores less favored areas.

→ PILAF is presented as a practical simplification of T-PILAF. PILAF avoids complex normalization factors and approximates policy interpolation token-wise.

→ PILAF samples responses from either the current policy π, or by interpolating logits of the current policy π and reference policy π_ref. This interpolation uses the KL regularization coefficient β.

-----

Key Insights 💡:

→ T-PILAF aligns the gradient of the reward model loss with the policy gradient of the true human preference objective. This alignment ensures that minimizing the reward model loss directly contributes to maximizing human values.

→ T-PILAF improves statistical efficiency. It aligns optimization with directions of greatest sensitivity in the human preference objective. This leads to more informative preference data and reduces training variance.

→ PILAF, derived from T-PILAF, inherits these theoretical benefits. It improves sample efficiency and performance in practical RLHF settings.

-----

Results 📊:

→ In iterative Direct Preference Optimization (DPO), PILAF achieves baseline reward levels with 40% less training time.

→ In online DPO, PILAF demonstrates a better Reward-KL trade-off. It achieves higher reward with lower KL divergence compared to Vanilla and Hybrid Sampling.

→ In robustness tests with an overfitted initial model, PILAF escapes suboptimal regions and achieves higher reward and lower KL divergence than Vanilla Sampling.

→ PILAF reduces annotation and computation costs by over 40% in iterative DPO while maintaining comparable performance.