Gaussian Gated Linear Networks

Code

• github.com/deepmind/deepmind-research/tree/master/gated_linear_networks
  Officialtf★ 0
• github.com/2023-MindSpore-4/Code12/tree/main/d2l/chapter_03_linear-networks
  mindspore★ 0
• github.com/MindCode-4/code-7/tree/main/gaussian_adaptive_attention
  mindspore★ 0

Abstract

We propose the Gaussian Gated Linear Network (G-GLN), an extension to the recently proposed GLN family of deep neural networks. Instead of using backpropagation to learn features, GLNs have a distributed and local credit assignment mechanism based on optimizing a convex objective. This gives rise to many desirable properties including universality, data-efficient online learning, trivial interpretability and robustness to catastrophic forgetting. We extend the GLN framework from classification to multiple regression and density modelling by generalizing geometric mixing to a product of Gaussian densities. The G-GLN achieves competitive or state-of-the-art performance on several univariate and multivariate regression benchmarks, and we demonstrate its applicability to practical tasks including online contextual bandits and density estimation via denoising.

Tasks

Reproductions