Multimodal representation

Abstract

This paper considers a multimodal representation that associates image features to text such that the conditional probability of the next word given past n words and image features is defined by a neural language model for image retrieval and text generation. By contrast to previous representations, our representation is learned to resolve the issue of data sparsity that has been a deteriorative cause for any neural language model in the evaluation. Specifically, we make use of Kneser-Ney smoothing and skip-gram techniques in order to integrate each of them to a multimodal neural language model, e.g., the Modality-biased Log-bilinear model. As a result, the prediction for the next word using the conditional probability is developed to produce better contextual consistency within one unit of each modality, i.e., one sentence or one image. On the other hand, the correspondence of image and text is also enhanced. The representation is validated on the IAPR TC-12 and Attribute Discovery datasets for image retrieval and text generation, demonstrating improved performance on perplexity and BLEU-n criteria and effective shared representation learning.