On the understanding of sharpness-aware minimization and its application: a perspective on escape efficiency and asymmetric valleys

Abstract

Sharpness-Aware Minimization (SAM) has emerged as a promising training scheme that leads to good generalization through finding flat minima. Despite its accomplishments in various fields, the existing theoretical understanding of SAM is far behind its successes. To extend the understanding of SAM, we theoretically analyze the SAM from two novel perspectives: escape efficiency and asymmetric valleys. First, we prove that SAM can escape a minimum faster than SGD. Hence the SAM can explore more minima than SGD and can converge to flatter minima by escaping minima where SGD would be stranded. Second, we show that SAM converges to a flatter region on asymmetric valleys than SGD and it leads to better generalization. Moreover, we prove that these effects are amplified by increasing the radius of inner maximization. Based on the proposed theory, we further study an efficient way to utilize SAM, Parsimonious SAM (PSAM), which uses SAM periodically in the early phase of training. Finally, on various architectures and datasets, we empirically verify that the proposed theory holds well in practice, and PSAM presents comparable performance to SAM while it requires only 65% of the computational cost of SAM.