Attention Mechanism with Energy-Friendly Operations

Abstract

Attention mechanism has become the dominant module in natural language processing models. It is computationally intensive and depends on massive power-hungry multiplications. In this paper, we rethink variants of attention mechanism from the energy consumption aspects. After reaching the conclusion that the energy costs of several energy-friendly operations are far less than their multiplication counterparts, we build a novel attention model by completely replacing multiplications with either selective operations or additions. Empirical results on three machine translation tasks demonstrate that the proposed method, against the vanilla one, achieves comparable accuracy while only consumes a half of energy. Our code will be released upon the acceptance.

Tasks

Reproductions