Equivalence of approximation by convolutional neural networks and fully-connected networks

Abstract

Convolutional neural networks are the most widely used type of neural networks in applications. In mathematical analysis, however, mostly fully-connected networks are studied. In this paper, we establish a connection between both network architectures. Using this connection, we show that all upper and lower bounds concerning approximation rates of fully-connected neural networks for functions f C -- for an arbitrary function class C -- translate to essentially the same bounds concerning approximation rates of convolutional neural networks for functions f C^equi, with the class C^equi consisting of all translation equivariant functions whose first coordinate belongs to C. All presented results consider exclusively the case of convolutional neural networks without any pooling operation and with circular convolutions, i.e., not based on zero-padding.

Tasks

Reproductions