Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples

Code

• github.com/google-research/meta-dataset
  OfficialIn papertf★ 798
• github.com/cambridge-mlg/cnaps
  pytorch★ 162
• github.com/peymanbateni/simple-cnaps
  pytorch★ 120
• github.com/mboudiaf/pytorch-meta-dataset
  pytorch★ 59
• github.com/plai-group/simple-cnaps
  tf★ 58
• github.com/nobody-1617/deta
  pytorch★ 17
• github.com/tmlr-group/CoPA
  tf★ 10
• github.com/jimzai/deta
  pytorch★ 6
• github.com/tmlr-group/mokd
  tf★ 4
• github.com/gokyeongryeol/MAHA
  pytorch★ 1

Abstract

Few-shot classification refers to learning a classifier for new classes given only a few examples. While a plethora of models have emerged to tackle it, we find the procedure and datasets that are used to assess their progress lacking. To address this limitation, we propose Meta-Dataset: a new benchmark for training and evaluating models that is large-scale, consists of diverse datasets, and presents more realistic tasks. We experiment with popular baselines and meta-learners on Meta-Dataset, along with a competitive method that we propose. We analyze performance as a function of various characteristics of test tasks and examine the models' ability to leverage diverse training sources for improving their generalization. We also propose a new set of baselines for quantifying the benefit of meta-learning in Meta-Dataset. Our extensive experimentation has uncovered important research challenges and we hope to inspire work in these directions.

Tasks

Benchmark Results

Reproductions