SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization

Code

• github.com/namisan/mt-dnn
  OfficialIn paperpytorch★ 2,257
• github.com/microsoft/MT-DNN
  pytorch★ 167
• github.com/archinetai/vat-pytorch
  pytorch★ 17
• github.com/cliang1453/camero
  pytorch★ 9
• github.com/chunhuililili/mt_dnn
  pytorch★ 2
• github.com/archinetai/smart-pytorch
  pytorch★ 0

Abstract

Transfer learning has fundamentally changed the landscape of natural language processing (NLP) research. Many existing state-of-the-art models are first pre-trained on a large text corpus and then fine-tuned on downstream tasks. However, due to limited data resources from downstream tasks and the extremely large capacity of pre-trained models, aggressive fine-tuning often causes the adapted model to overfit the data of downstream tasks and forget the knowledge of the pre-trained model. To address the above issue in a more principled manner, we propose a new computational framework for robust and efficient fine-tuning for pre-trained language models. Specifically, our proposed framework contains two important ingredients: 1. Smoothness-inducing regularization, which effectively manages the capacity of the model; 2. Bregman proximal point optimization, which is a class of trust-region methods and can prevent knowledge forgetting. Our experiments demonstrate that our proposed method achieves the state-of-the-art performance on multiple NLP benchmarks.

Tasks

Benchmark Results

Reproductions