Gibbs randomness-compression proposition: An efficient deep learning

Code

• github.com/msuzen/research/tree/main/gibbs-randomness-compression
  Officialnone★ 0

Abstract

A proposition that connects randomness and compression put forward via Gibbs entropy over set of measurement vectors associated with a compression process. The proposition states that a lossy compression process is equivalent to directed randomness that preserves information content. The proposition originated from the observed behaviour in newly proposed Dual Tomographic Compression (DTC) compress-train framework. This is akin to tomographic reconstruction of layer weight matrices via building compressed sensed projections, so called weight rays. This tomographic approach is applied to previous and next layers in a dual fashion, that triggers neuronal-level pruning. This novel model compress-train scheme appear in iterative fashion and act as smart neural architecture search, Experiments demonstrated utility of this dual-tomography producing state-of-the-art performance with efficient compression during training, accelerating and supporting lottery ticket hypothesis. However, random compress-train iterations having similar performance demonstrated the connection between randomness and compression from statistical physics perspective, we formulated so called Gibbs randomness-compression proposition, signifying randomness-compression relationship via Gibbs entropy. Practically, DTC framework provides a promising approach for massively energy and resource efficient deep learning training approach.

Tasks

Reproductions