Concurrent training methods for Kolmogorov-Arnold networks: Disjoint datasets and FPGA implementation

Abstract

The present paper introduces concurrency-driven enhancements to the training algorithm for the Kolmogorov-Arnold networks (KANs) that is based on the Newton-Kaczmarz (NK) method. As indicated by prior research, KANs trained using the NK-based approach significantly overtake classical neural networks based on multilayer perceptrons (MLPs) in terms of accuracy and training time. Although certain parts of the algorithm, such as evaluation of the basis functions, can be parallelised, a fundamental limitation lies in the computation of the updates' values that is sequential: each update depends on the results of the previous step, obstructing parallel application. However, substantial acceleration is achievable. Three complementary strategies are proposed in the present paper: (i) a pre-training procedure tailored to the NK updates' structure, (ii) training on disjoint subsets of data, followed by models' merging, not in the context of federated learning, but as a mechanism for accelerating the convergence, and (iii) a parallelisation technique suitable for execution on field-programmable gate arrays (FPGAs), which is implemented and tested directly on the device. All experimental results presented in this work are fully reproducible, with the complete source codes available online.

Reproductions