Applying SoftTriple Loss for Supervised Language Model Fine Tuning

Abstract

We introduce a new loss function TripleEntropy to improve classification performance for fine-tuninggeneral knowledge pre-trained language models based on cross-entropy and SoftTriple loss. Thisloss function can improve the robust RoBERTa baseline model fine-tuned with cross-entropy loss byabout (0.02% - 2.29%). Thorough tests on popular datasets indicate a steady gain. The fewer samplesin the training dataset, the higher gain – thus, for small-sized dataset it is 0.78%, for medium-sized –0.86% for large – 0.20% and for extra-large 0.04%.

Tasks

Reproductions