Text Summarization with Pretrained Encoders

Code

• github.com/nlpyang/PreSumm
  OfficialIn paperpytorch★ 1,302
• github.com/HHousen/TransformerSum
  pytorch★ 439
• github.com/nakhunchumpolsathien/TR-TPBS
  none★ 29
• github.com/alebryvas/berk266
  pytorch★ 20
• github.com/nachotp/BertCommentSum
  pytorch★ 13
• github.com/olivia-fsm/p2mcq
  pytorch★ 10
• github.com/chesterdu/contrastive_summary
  pytorch★ 3
• github.com/manshri/tesum
  pytorch★ 1
• github.com/raqoon886/KoBertSum
  pytorch★ 0
• github.com/raqoon886/KorBertSum
  pytorch★ 0

Abstract

Bidirectional Encoder Representations from Transformers (BERT) represents the latest incarnation of pretrained language models which have recently advanced a wide range of natural language processing tasks. In this paper, we showcase how BERT can be usefully applied in text summarization and propose a general framework for both extractive and abstractive models. We introduce a novel document-level encoder based on BERT which is able to express the semantics of a document and obtain representations for its sentences. Our extractive model is built on top of this encoder by stacking several inter-sentence Transformer layers. For abstractive summarization, we propose a new fine-tuning schedule which adopts different optimizers for the encoder and the decoder as a means of alleviating the mismatch between the two (the former is pretrained while the latter is not). We also demonstrate that a two-staged fine-tuning approach can further boost the quality of the generated summaries. Experiments on three datasets show that our model achieves state-of-the-art results across the board in both extractive and abstractive settings. Our code is available at https://github.com/nlpyang/PreSumm

Tasks

Benchmark Results

Reproductions