In Search of Lost Domain Generalization

Code

• github.com/facebookresearch/DomainBed
  OfficialIn paperpytorch★ 1,604
• github.com/YugeTen/fish
  pytorch★ 124
• github.com/weixin-liang/metashift
  pytorch★ 108
• github.com/matsuolab/t3a
  pytorch★ 106
• github.com/inouye-lab/feddg_benchmark
  pytorch★ 27
• github.com/facebookresearch/domainbed_measures
  pytorch★ 18
• github.com/nancyquris/xdomainmix
  pytorch★ 18
• github.com/wangxinyilinda/causal-balancing-for-domain-generalization
  pytorch★ 13
• github.com/qiaoruiyt/noiserobustdg
  pytorch★ 5
• github.com/kowshikthopalli/DREAME
  pytorch★ 2

Abstract

The goal of domain generalization algorithms is to predict well on distributions different from those seen during training. While a myriad of domain generalization algorithms exist, inconsistencies in experimental conditions -- datasets, architectures, and model selection criteria -- render fair and realistic comparisons difficult. In this paper, we are interested in understanding how useful domain generalization algorithms are in realistic settings. As a first step, we realize that model selection is non-trivial for domain generalization tasks. Contrary to prior work, we argue that domain generalization algorithms without a model selection strategy should be regarded as incomplete. Next, we implement DomainBed, a testbed for domain generalization including seven multi-domain datasets, nine baseline algorithms, and three model selection criteria. We conduct extensive experiments using DomainBed and find that, when carefully implemented, empirical risk minimization shows state-of-the-art performance across all datasets. Looking forward, we hope that the release of DomainBed, along with contributions from fellow researchers, will streamline reproducible and rigorous research in domain generalization.

Tasks

Benchmark Results

Reproductions