Pixel-level matching for video object segmentation using convolutional neural networks

Abstract

In this thesis, we propose a novel video object segmentation algorithm based on pixel-level matching using Convolutional Neural Networks (CNN). Our network architecture combines a generative model with a discriminative one. The generative Siamese structure encodes the pixel-level similarity between query and search inputs. A target objectness is then discriminated from the background by decoding the matching scores. The feature instances exploited for similarity encoding are computed and compressed from multiple layers with different depths to take advantages of both spatial details and semantic information. Thanks to the feature compression technique, we can lighten the network and boost the computational efficiency (about 8 ms for one feed forward). In the online sequence, all the frames are matched with the initial query frame using fine-tuned pixel-level matching network. To the best of our knowledge, this is the first approach targeting mask propagation using a deep learning network. Experiments on large datasets demonstrate the effectiveness of our combinative model achieving state-of-the-art results. In addition, we introduce the transferability of our network to the different domain such as infrared data. Finally, the applicability of our network is validated through the newly designed video stabilization and co-segmentation.