KTO: Model Alignment as Prospect Theoretic Optimization

Code

• github.com/contextualai/halos
  OfficialIn paperpytorch★ 907
• github.com/openbmb/eurus
  pytorch★ 321
• github.com/tidedra/vl-rlhf
  pytorch★ 197
• github.com/reml-group/deliberation-on-priors
  pytorch★ 38
• github.com/uppaal/detox-edit
  pytorch★ 6

Abstract

Kahneman & Tversky's prospect theory tells us that humans perceive random variables in a biased but well-defined manner (1992); for example, humans are famously loss-averse. We show that objectives for aligning LLMs with human feedback implicitly incorporate many of these biases -- the success of these objectives (e.g., DPO) over cross-entropy minimization can partly be ascribed to them belonging to a family of loss functions that we call human-aware losses (HALOs). However, the utility functions these methods attribute to humans still differ from those in the prospect theory literature. Using a Kahneman-Tversky model of human utility, we propose a HALO that directly maximizes the utility of generations instead of maximizing the log-likelihood of preferences, as current methods do. We call this approach KTO, and it matches or exceeds the performance of preference-based methods at scales from 1B to 30B, despite only learning from a binary signal of whether an output is desirable. More broadly, our work suggests that there is no one HALO that is universally superior; the best loss depends on the inductive biases most appropriate for a given setting, an oft-overlooked consideration.

Tasks

Reproductions