BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Elad Ben Zaken, Shauli Ravfogel, Yoav Goldberg
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/benzakenelad/BitFitOfficialIn paperpytorch★ 143
- github.com/mkshing/DiffFit-pytorchpytorch★ 95
- github.com/uds-lsv/llmftpytorch★ 31
- github.com/Aradhye2002/selective-peft-toolkitpytorch★ 9
- github.com/cloudygoose/fewshot_lamapytorch★ 7
- github.com/piero2c/BitInferpytorch★ 2
Abstract
We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.