Continual learning with hypernetworks

Code

• github.com/chrhenning/hypercl
  OfficialIn paperpytorch★ 0
• github.com/chrhenning/hypnettorch
  pytorch★ 133
• github.com/rvl-lab-utoronto/HyperCRL
  pytorch★ 13
• github.com/gahaalt/continual-learning-overview
  tf★ 11
• github.com/pennfranc/hypnettorch
  pytorch★ 2
• github.com/gmum/hyperinterval
  pytorch★ 1
• github.com/gmum/hint
  pytorch★ 1
• github.com/gahaalt/continual-learning-with-hypernets
  tf★ 0
• github.com/geox-lab/cmn
  pytorch★ 0

Abstract

Artificial neural networks suffer from catastrophic forgetting when they are sequentially trained on multiple tasks. To overcome this problem, we present a novel approach based on task-conditioned hypernetworks, i.e., networks that generate the weights of a target model based on task identity. Continual learning (CL) is less difficult for this class of models thanks to a simple key feature: instead of recalling the input-output relations of all previously seen data, task-conditioned hypernetworks only require rehearsing task-specific weight realizations, which can be maintained in memory using a simple regularizer. Besides achieving state-of-the-art performance on standard CL benchmarks, additional experiments on long task sequences reveal that task-conditioned hypernetworks display a very large capacity to retain previous memories. Notably, such long memory lifetimes are achieved in a compressive regime, when the number of trainable hypernetwork weights is comparable or smaller than target network size. We provide insight into the structure of low-dimensional task embedding spaces (the input space of the hypernetwork) and show that task-conditioned hypernetworks demonstrate transfer learning. Finally, forward information transfer is further supported by empirical results on a challenging CL benchmark based on the CIFAR-10/100 image datasets.

Tasks

Benchmark Results

Reproductions