Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Code

• github.com/UKPLab/sentence-transformers
  OfficialIn paperpytorch★ 0
• github.com/princeton-nlp/SimCSE
  pytorch★ 3,646
• github.com/InsaneLife/dssm
  tf★ 667
• github.com/law-ai/summarization
  pytorch★ 220
• github.com/valdecy/pybibx
  tf★ 195
• github.com/BinWang28/BERT_Sentence_Embedding
  pytorch★ 182
• github.com/BinWang28/SBERT-WK-Sentence-Embedding
  pytorch★ 182
• github.com/BM-K/KoSentenceBERT_ETRI
  pytorch★ 162
• github.com/BM-K/KoSentenceBERT
  pytorch★ 162
• github.com/BM-K/KoSentenceBERT_SKT
  pytorch★ 142

Abstract

BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering. In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.

Tasks

Benchmark Results

Reproductions