Robust Bi-Tempered Logistic Loss Based on Bregman Divergences
Ehsan Amid, Manfred K. Warmuth, Rohan Anil, Tomer Koren
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/google/bi-tempered-lossOfficialIn papertf★ 0
- github.com/mlpanda/bi-tempered-loss-pytorchpytorch★ 61
- github.com/MindSpore-scientific-2/code-5/tree/main/bi-tempered-loss-pytorchmindspore★ 0
- github.com/NingAnMe/Bi-Tempered-Logistic-Loss-PyTorchpytorch★ 0
- github.com/MindSpore-scientific-2/code-4/tree/main/bi-tempered-loss-pytorchmindspore★ 0
- github.com/MindSpore-scientific-2/code-9/tree/main/bi-tempered-loss-pytorchmindspore★ 0
- github.com/pwc-1/Paper-9/tree/main/4/bi-tempered-loss-pytorchmindspore★ 0
- github.com/prakashsellathurai/cassava-leaf-disease-classificationpytorch★ 0
- github.com/Diulhio/bitemperedloss-tftf★ 0
- github.com/MindSpore-scientific/code-4/tree/main/bi-tempered-loss-pytorchmindspore★ 0
Abstract
We introduce a temperature into the exponential function and replace the softmax output layer of neural nets by a high temperature generalization. Similarly, the logarithm in the log loss we use for training is replaced by a low temperature logarithm. By tuning the two temperatures we create loss functions that are non-convex already in the single layer case. When replacing the last layer of the neural nets by our bi-temperature generalization of logistic loss, the training becomes more robust to noise. We visualize the effect of tuning the two temperatures in a simple setting and show the efficacy of our method on large data sets. Our methodology is based on Bregman divergences and is superior to a related two-temperature method using the Tsallis divergence.