Continual learning using image-conditional prompt for vision-language pre-training

Abstract

Deep learning has achieved remarkable performance in several areas, but fine-tuning pre-trained models based on new data can lead to catastrophic forgetting. To address this, we propose a novel image conditional prompt learning approach to continual learning method which is inspired by humans’ working memory system. Our approach eliminates the need for data storage buffers and prompt pools for continual learning. Instead, we just need to train only the lightweight MLP to generate prompts without training the entire model. Leveraging CLIP-based models allows us to align vision and text, facilitating comprehensive multi-modal learning. Also, our approach uses regularization and knowledge distillation to retain knowledge while adapting to new tasks.