Algorithm / hardware co-design for adversarial training acceleration through selective computations

Abstract

The robustness of neural networks has been emphasized since some works discovered that small perturbations could cause misclassification. Various defense methods against adversarial attacks have been proposed, and adversarial training is one of the most effective. Adversarial training generates adversarial examples and learns such data so that models can classify adversarial examples accurately. Multiple steps of forward pass and backpropagation are necessary to generate adversarial examples, and it consumes enormous computation costs and latency. This work accelerates adversarial training by selective computation of features of which gradients are large. This work also proposes a fast gradient top-k method exploiting the distribution of exponents and selectively computes large gradients in FP32. Efficient hardware to support selective computation, exponent approximation, and gradient top-k is designed with a small overhead.