LongT5: Efficient Text-To-Text Transformer for Long Sequences

Code

• github.com/google-research/longt5
  OfficialIn papertf★ 184
• github.com/utsjiyaoli/qa-attack
  pytorch★ 5
• github.com/MindSpore-scientific-2/code-14/tree/main/longt5
  mindspore★ 0
• github.com/pwc-1/Paper-9/tree/main/longt5
  mindspore★ 0

Abstract

Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the performance of Transformer-based neural models. In this paper, we present a new model, called LongT5, with which we explore the effects of scaling both the input length and model size at the same time. Specifically, we integrated attention ideas from long-input transformers (ETC), and adopted pre-training strategies from summarization pre-training (PEGASUS) into the scalable T5 architecture. The result is a new attention mechanism we call Transient Global (TGlobal), which mimics ETC's local/global attention mechanism, but without requiring additional side-inputs. We are able to achieve state-of-the-art results on several summarization tasks and outperform the original T5 models on question answering tasks.

Tasks

Benchmark Results

Reproductions