DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models

Code

• github.com/zjuruizhechen/diffpo
  Official★ 9

Abstract

Inference-time alignment provides an efficient alternative for aligning LLMs with humans. However, these approaches still face challenges, such as limited scalability due to policy-specific value functions and latency during the inference phase. In this paper, we propose a novel approach, Diffusion-styled Preference Optimization ( ), which provides an efficient and policy-agnostic solution for aligning LLMs with humans. By directly performing alignment at sentence level, ~avoids the time latency associated with token-level generation. Designed as a plug-and-play module, ~can be seamlessly integrated with various base models to enhance their alignment. Extensive experiments on AlpacaEval 2, MT-bench, and HH-RLHF demonstrate that ~achieves superior alignment performance across various settings, achieving a favorable trade-off between alignment quality and inference-time latency. Furthermore, ~demonstrates model-agnostic scalability, significantly improving the performance of large models such as Llama-3-70B.

Tasks

Reproductions