HBoP: Hierarchical bag of phrases

Abstract

Visual-Language Models (VLMs) play a crucial role in connecting the gap that exists between understanding visual and linguistic data collectively. However, the success of current models is hindered by the extensive pre-training and fine-tuning required, often making them difficult to employ for downstream tasks. To address this limitation, large language models were introduced as an alternative to fine-tuning VLMs due to their zero-shot applicability in downstream tasks, but the effective utilization of LLMs for vision-language tasks demands comprehensive textual representations of the visual data in the form of captions. Unfortunately, the textual representations generated by current VLMs are repetitive and do not provide a detailed understanding of the data. To address this gap, we propose a novel framework, Hierarchical Bag of Phrases (HBoP), that effectively connects visual and textual data by generating a comprehensive understanding of all pertinent information in the image. Our proposed framework not only enables the use of LLMs in multi-modal tasks but also helps produce image-patch/text pairs that could be useful for training vision-language models for better image representation. To evaluate its performance, we conduct experiments comparing HBoP results to state-of-the-art VLMs in terms of semantic integrity, image-text retrieval, and the diversity of generated captions. Our results demonstrate a diversity score significantly close to human-generated captions and a substantial increase in performance for text-retrieval tasks, showcasing the effectiveness of the HBoP framework.