IVIST: Interactive Video Search Tool in VBS 2022

Abstract

This paper presents the details of the proposed video retrieval tool, named Interactive VIdeo Search Tool (IVIST) for the Video Browser Showdown (VBS) 2022. In order to retrieve desired videos from a multimedia database, it is necessary to match queries from humans and video shots in the database effectively. To boost such matching relationship, we propose a multi-modal-based retrieval scheme that can fully utilize various modal features of the multimedia data and synthetically consider the matching relationships between modalities. The proposed IVIST maps human-made queries (e.g., language) and features (e.g., visual and sound) from the database into a multi-modal matching latent space through deep neural networks. Based on the latent space, videos with high similarity to the query feature are suggested as candidate shots. Prior knowledge-based filtering can be further applied to refine the results of candidate shots. Moreover, the user interface of the tool is devised in a user-friendly way for interactive video searching.