TinyBERT: Distilling BERT for Natural Language Understanding
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, Qun Liu
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/huawei-noah/Pretrained-Language-Model/tree/master/TinyBERTOfficialtf★ 0
- github.com/graison-thomas/TinyFinBERTnone★ 2
- github.com/2023-MindSpore-1/ms-code-166mindspore★ 0
- github.com/pwc-1/Paper-9/tree/main/1/tinybertmindspore★ 0
- github.com/mindspore-ai/models/tree/master/official/nlp/tinybertmindspore★ 0
- github.com/pwc-1/Paper-10/tree/main/tinybertmindspore★ 0
- github.com/mkavim/finetune_berttf★ 0
- github.com/PaddlePaddle/PaddleNLP/tree/develop/paddlenlp/transformers/tinybertpaddle★ 0
- github.com/MindCode-4/code-5/tree/main/tinybertmindspore★ 0
- github.com/xiaolilaoli/tiny_bert_msmindspore★ 0
Abstract
Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually computationally expensive, so it is difficult to efficiently execute them on resource-restricted devices. To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models. By leveraging this new KD method, the plenty of knowledge encoded in a large teacher BERT can be effectively transferred to a small student Tiny-BERT. Then, we introduce a new two-stage learning framework for TinyBERT, which performs Transformer distillation at both the pretraining and task-specific learning stages. This framework ensures that TinyBERT can capture he general-domain as well as the task-specific knowledge in BERT. TinyBERT with 4 layers is empirically effective and achieves more than 96.8% the performance of its teacher BERTBASE on GLUE benchmark, while being 7.5x smaller and 9.4x faster on inference. TinyBERT with 4 layers is also significantly better than 4-layer state-of-the-art baselines on BERT distillation, with only about 28% parameters and about 31% inference time of them. Moreover, TinyBERT with 6 layers performs on-par with its teacher BERTBASE.