Convolutional Neural Network With Developmental Memory for Continual Learning

Abstract

Convolutional neural networks (CNNs) are one of the most successful deep neural networks. Indeed, most of the recent applications related to computer vision are based on CNNs. However, when learning new tasks in a sequential manner, CNNs face catastrophic forgetting: they forget a considerable amount of previously learned tasks while adapting to novel tasks. To overcome this main barrier to continual learning with CNNs, we introduce developmental memory (DM) into a CNN, continually generating submemory networks to learn important features of individual tasks. A novel training method, referred to here as guided learning (GL), guides the newly generated submemory to become an expert on the new task, eventually improving the performance of the overall network. At the same time, the existing submemories attempt to preserve the knowledge of old tasks. Experiments on image classification tasks show that compared with the state-of-the-art algorithms, the proposed CNN with DM not only improves the classification performance on the new image task but also leads to less forgetting of previous image tasks to facilitate continual learning.