Ada-boundary: accelerating DNN training via adaptive boundary batch selection

Abstract

Neural networks converge faster with help from a smart batch selection strategy. In this regard, we proposeAda-Boundary, a novel and simple adaptive batch selection algorithm that constructs an effective mini-batch according to the learning progress of the model. Our key idea is to exploitconfusingsamples for which the model cannot predict labels with high confidence. Thus, samples near the current decision boundary are considered to be the most effective for expediting convergence. Taking advantage of this design,Ada-Boundarymaintained its dominance for various degrees of training difficulty. We demonstrate the advantage ofAda-Boundaryby extensive experimentation using CNNs with five benchmark data sets.Ada-Boundarywas shown to produce a relative improvement in test errors by up to 31.80% compared with the baseline for a fixed wall-clock training time, thereby achieving a faster convergence speed.