How to Train BERT with an Academic Budget

Code

• github.com/IntelLabs/academic-budget-bert
  OfficialIn paperpytorch★ 0
• github.com/peteriz/academic-budget-bert
  OfficialIn paperpytorch★ 0
• github.com/octanove/shiba
  pytorch★ 89
• github.com/yxzwang/normalized-information-payload
  pytorch★ 9

Abstract

While large language models a la BERT are used ubiquitously in NLP, pretraining them is considered a luxury that only a few well-funded industry labs can afford. How can one train such models with a more modest budget? We present a recipe for pretraining a masked language model in 24 hours using a single low-end deep learning server. We demonstrate that through a combination of software optimizations, design choices, and hyperparameter tuning, it is possible to produce models that are competitive with BERT-base on GLUE tasks at a fraction of the original pretraining cost.

Tasks

Benchmark Results

Reproductions