Probabilistic Transformers

Code

• github.com/apple/ml-probabilistic-attention
  pytorch★ 33

Abstract

We show that Transformers are Maximum Posterior Probability estimators for Mixtures of Gaussian Models. This brings a probabilistic point of view to Transformers and suggests extensions to other probabilistic cases.

Reproductions