SWALP : Stochastic Weight Averaging in Low-Precision Training
Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, Christopher De Sa
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/stevenygd/SWALPOfficialIn paperpytorch★ 0
- github.com/RICE-EIC/Early-Bird-Ticketspytorch★ 140
- github.com/gatech-eic/early-bird-ticketspytorch★ 0
Abstract
Low precision operations can provide scalability, memory savings, portability, and energy efficiency. This paper proposes SWALP, an approach to low precision training that averages low-precision SGD iterates with a modified learning rate schedule. SWALP is easy to implement and can match the performance of full-precision SGD even with all numbers quantized down to 8 bits, including the gradient accumulators. Additionally, we show that SWALP converges arbitrarily close to the optimal solution for quadratic objectives, and to a noise ball asymptotically smaller than low precision SGD in strongly convex settings.