Disjoint multi-task learning between heterogeneous action and caption data

Abstract

It is believed that the success of deep neural networks on various image tasks is achieved by virtue of a large number of annotated data. When it comes to video related tasks, while there have been various datasets, the number of annotated videos in a single dataset is still far less than that of image datasets. In this paper, we leverage existing video datasets that have heterogeneous videos and annotations, so that a model can be trained while compensating for the limit of a single dataset size. Since the video data in each dataset has heterogeneous annotations, traditional multi-task learning is not available in this scenario. To this end, we propose a simple alternating directional optimization method to efficiently learn from the heterogeneous data. We demonstrate the effectiveness of our model on both action recognition and caption embedding tasks. With our method, we show performance improvements on action recognition task and comparable performance on sentence retrieval task to the model trained on a single-task data.