On Calibration of Modern Neural Networks

Code

• github.com/gpleiss/temperature_scaling
  OfficialIn paperpytorch★ 0
• github.com/aigen/df-posthoc-calibration
  none★ 41
• github.com/anonwhymoos/connectivity
  jax★ 18
• github.com/sleep3r/garrus
  none★ 3
• github.com/johntd54/stanford_car
  pytorch★ 0
• github.com/AnanyaKumar/verified_calibration
  tf★ 0
• github.com/sirius8050/Expected-Calibration-Error
  none★ 0
• github.com/saurabhgarg1996/calibration
  pytorch★ 0
• github.com/Andreas12321/Est-Cert-Final
  tf★ 0
• github.com/cpark321/bayesian-neural-networks
  pytorch★ 0

Abstract

Confidence calibration -- the problem of predicting probability estimates representative of the true correctness likelihood -- is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-of-the-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.

Tasks

Reproductions