Training Multi-Layer Binary Neural Networks With Local Binary Error Signals

Abstract

Binary Neural Networks (BNNs) significantly reduce computational complexity and memory usage in machine and deep learning by representing weights and activations with just one bit. However, most existing training algorithms for BNNs rely on quantization-aware floating-point Stochastic Gradient Descent (SGD), limiting the full exploitation of binary operations to the inference phase only. In this work, we propose, for the first time, a fully binary and gradient-free training algorithm for multi-layer BNNs, eliminating the need for back-propagated floating-point gradients. Specifically, the proposed algorithm relies on local binary error signals and binary weight updates, employing integer-valued hidden weights that serve as a synaptic metaplasticity mechanism, thereby enhancing its neurobiological plausibility. The fully binary and gradient-free algorithm introduced in this paper enables the training of binary multi-layer perceptrons with binary inputs, weights, and activations, by using exclusively XNOR, Popcount, and increment/decrement operations. Experimental results on multi-class classification benchmarks show test accuracy improvements of up to +35.47% over the only existing fully binary single-layer state-of-the-art solution. Compared to full-precision SGD, our solution improves test accuracy by up to +41.31% under the same total memory demandx2013including the model, activations, and input datasetx2013while also reducing computational cost by two orders of magnitude in terms of the total number of equivalent Boolean gates. The proposed algorithm is made available to the scientific community as a public repository.

Tasks

Reproductions