Universal Language Model Fine-tuning for Text Classification

Code

• github.com/fastai/fastai
  Officialpytorch★ 27,931
• github.com/mrdbourke/tensorflow-deep-learning
  tf★ 5,869
• github.com/Deepayan137/Adapting-OCR
  pytorch★ 62
• github.com/tanmaylaud/Patient_Conversation_Classifier_FastAI
  none★ 1
• github.com/comicencyclo/TransferLearning_DiscriminativeFineTuning
  none★ 1
• github.com/amagooda/SummaRuNNer_coattention
  pytorch★ 1
• github.com/apmoore1/language-model
  pytorch★ 1
• github.com/PrideLee/sentiment-analysis
  pytorch★ 0
• github.com/castortroynz/desafio_atuacao19
  none★ 0
• github.com/Socialbird-AILab/BERT-Classification-Tutorial
  tf★ 0

Abstract

Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce techniques that are key for fine-tuning a language model. Our method significantly outperforms the state-of-the-art on six text classification tasks, reducing the error by 18-24% on the majority of datasets. Furthermore, with only 100 labeled examples, it matches the performance of training from scratch on 100x more data. We open-source our pretrained models and code.

Tasks

Benchmark Results

Reproductions