Toward universal computer vision task solver with single unified model

Abstract

With the advancements in Large Language Models (LLMs), a variety of Natural Language Processing(NLP) tasks can be effectively addressed using single unified LLM backbones. Notably, Instruction Tuning leverages the emergent abilities of LLMs by handling diverse language tasks through language instructions. However, in the field of computer vision, there is no single unified system capable of solving all types of computer vision tasks due to the inherent diversity of such tasks. In this paper, we propose an approach to address various computer vision tasks by utilizing the capabilities of visual instruction tuning. By unifying the model’s input and output as either text or image, we design a sequence-to- sequence modeling framework for computer vision tasks. In summary, we present a framework designed to solve any type of computer vision task—a universal computer vision task solver