BlitzRank: Principled Zero-shot Ranking Agents with Tournament Graphs

Code

• github.com/contextualai/blitzrank
  OfficialIn paper★ 0

Abstract

Selecting the top m from n items via expensive k-wise comparisons is central to settings ranging from LLM-based document reranking to crowdsourced evaluation and tournament design. Existing methods either rely on heuristics that discard comparison information, or exploit it at prohibitive cost. We introduce a tournament graph framework that provides a principled foundation for k-wise ranking. Our key observation is that each k-item comparison reveals an induced tournament of k2 pairwise preferences; aggregating these into a global preference graph and computing its transitive closure yields many additional orderings without further oracle calls. We formalize when the current top-m output is certifiably determined and design a greedy query schedule that maximizes information gain towards identifying the top-m items. The framework also gracefully handles non-transitive preferences -- cycles induced by real-world oracles -- by collapsing them into equivalence classes that yield principled tiered rankings. Applied to LLM reranking across 14 benchmarks and 5 models, BlitzRank achieves Pareto dominance over existing approaches: matching or exceeding accuracy while requiring 25--40% fewer tokens than comparable methods; against pairwise reranking, it achieves near-identical quality with 7 fewer tokens. Code available at https://github.com/ContextualAI/BlitzRank.

Reproductions