Local Linear Approximation Algorithm for Neural Network

Abstract

This paper is concerned with estimation of weights and biases in feed forward neural network (FNN). We propose using local linear approximation (LLA) for the activation function, and develop a LLA algorithm to estimate the weights and biases of one hidden layer FNN by iteratively linear regression. We further propose the layerwise optimized adaptive neural network (LOAN), in which we use the LLA to estimate the weights and biases in the LOAN layer by layer adaptively. We compare the performance of the LOAN with the commonly-used procedures in deep learning via analyses of four benchmark data sets. The numerical comparison implies that the proposed LOAN may outperform the existing procedures.

Tasks

Reproductions