Development and application of efficient data pruning techniques in deep learning

Abstract

Recent advancements in deep learning technology have brought innovations across various fields. However, as the technology has evolved towards using more data and larger models for performance improvement, there has been an exponential increase in the required computational costs. Consequently, the importance of efficient learning techniques, especially in data pruning, is becoming increasingly significant. Nevertheless, there are two key issues with existing data pruning methodologies: the necessity of training with the entire dataset for data selection and the variance in each methodology's performance depending on the data selection ratio. This research proposes methodologies to address these two critical issues in data selection. To tackle the first issue, we propose a `CG-score' (Complexity Gap score), which allows for the understanding of data characteristics without training, and demonstrate that the data selection performance based on this score is comparable to that of existing methodologies. By utilizing the Neural Tangent Kernel, which can mathematically approximate the learning process without directly training deep learning models, we quantified the characteristics of the data using only the training data. For the second issue, we proposed a `BWS' (best window selection) methodology, which involves sorting data by difficulty score and adjusting the selection range according to the selection ratio. We theoretically verify that changing the selection region according to different ratios enables optimal data selection and empirically confirm that this approach outperforms existing methodologies across all selection ratios.