Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring

Code

• github.com/sfzhou5678/PolyEncoder
  pytorch★ 249
• github.com/chijames/Poly-Encoder
  pytorch★ 167
• github.com/csong27/collision-bert
  pytorch★ 25
• github.com/fangrouli/Document-embedding-generation-models
  pytorch★ 3
• github.com/llStringll/Poly-encoders
  pytorch★ 0
• github.com/i2r-simmc/i2r-simmc-2020
  pytorch★ 0
• github.com/Alexey-Borisov/3_course_diary
  none★ 0

Abstract

The use of deep pre-trained bidirectional transformers has led to remarkable progress in a number of applications (Devlin et al., 2018). For tasks that make pairwise comparisons between sequences, matching a given input with a corresponding label, two approaches are common: Cross-encoders performing full self-attention over the pair and Bi-encoders encoding the pair separately. The former often performs better, but is too slow for practical use. In this work, we develop a new transformer architecture, the Poly-encoder, that learns global rather than token level self-attention features. We perform a detailed comparison of all three approaches, including what pre-training and fine-tuning strategies work best. We show our models achieve state-of-the-art results on three existing tasks; that Poly-encoders are faster than Cross-encoders and more accurate than Bi-encoders; and that the best results are obtained by pre-training on large datasets similar to the downstream tasks.

Tasks

Benchmark Results

Reproductions