On the connection between lottery ticket hypothesis and neural tangent Kernel

Abstract

Pruning is a technique that removes unnecessary weights from the neural network to obtain a memory efficient subnetwork with comparable accuracy to the original network. Lottery Ticket Hypothesis conjectures that most randomly-initialized, dense networks have subnetworks that match accuracy of the original networks when trained in isolation, and Iterative Magnitude Pruning (IMP) is the algorithm typically used to find such subnetworks. We conjecture that the Neural Tangnet Kernel (NTK) theory may be used to analyze Lottery Ticket Hypothesis and the IMP algorithm. We give an intuition about the conjecture, and provide an experimental evidence for our conjecture using the MNIST dataset. We also suggest a novel pruning scheme based on the conjecture.