Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want

Abstract

The interaction between humans and artificial intelligence (AI) is a crucial factor that reflects the effectiveness of multimodal large language models (MLLMs). However, current MLLMs primarily focus on image-level comprehension and limit interaction to textual instructions, thereby constraining their flexibility in usage and depth of response. In this paper, we introduce the Draw-and-Understand project: a new model, a multi-domain dataset, and a challenging benchmark for visual prompting. Specifically, we propose SPHINX-V, a new end-to-end trained Multimodal Large Language Model (MLLM) that connects a vision encoder, a visual prompt encoder and an LLM for various visual prompts (points, bounding boxes, and free-form shape) and language understanding. To advance visual prompting research for MLLMs, we introduce MDVP-Data and MDVP-Bench. MDVP-Data features a multi-domain dataset containing 1.6M unique image-visual prompt-text instruction-following samples, including natural images, document images, OCR images, mobile screenshots, web screenshots, and multi-panel images. Furthermore, we present MDVP-Bench, a comprehensive and challenging benchmark to assess a model’s capability in understanding visual prompting instructions. Our experiments demonstrate SPHINX-V’s impressive multimodal interaction capabilities through visual prompting, revealing significant improvements in detailed pixel-level description and question-answering abilities.

Figure 1: An Overview of the Draw-and-Understand Project. (a) Illustrating the task of pixel-level referring understanding, where visual prompts and an input image are processed by the model to generate a targeted response. (b) The architecture of SPHINX-V, which consists of an image encoder, a visual prompt encoder, and an LLM. (c) The data generation process for training, which involves two components: reconstruction of open-source data and data generation assisted by GPT-4V.

1 Introduction

Recent works [35, 78, 33, 2, 34] have enhanced Multimodal Large Language Models (MLLMs) with visual perception, facilitating image-related communication and a deeper understanding of the world. However, current MLLMs tend to focus on comprehending entire images by aligning them with textual information at the image level. This method restricts users from highlighting specific areas within images for more detailed inquiries. Therefore, enabling models to understand at the pixel level presents a significant research challenge.

To meet this challenge, recent advancements have concentrated on employing visual prompting to enhance pixel-level comprehension. Specifically, research works [74, 6, 7] utilize textual representations to delineate coordinates within images, thereby enhancing interactions with models. Another group of studies [46, 73, 77] employ positional embeddings to achieve enhanced spatial recognition. Meanwhile, efforts by [51, 73, 67, 70, 4] focus on Regions of Interest (ROI) to augment focus on specific areas within images. Each of these methods has shown promising results, yet they come with their respective limitations. ROI-based methods, for instance, rely on fine-grained masks for precise object representation. However, they often require pre-existing segmentation models or externally provided ground truth masks, which limits user flexibility and model scalability. Furthermore, some methods rely on fixed-format visual prompts such as bounding boxes, which are not user-friendly. A critical shortfall is the inability of most existing methods to support referring to multiple objects simultaneously. This limitation restricts usage flexibility and hampers the achievement of more complex understandings, such as discerning the nuanced interrelations and spatial dynamics among surrounding entities and the backdrop.

To address these shortcomings, we introduce SPHINX-V, an end-to-end multimodal large language model designed to enhance pixel-level image understanding. As depicted in Fig.1(b), the SPHINX-V architecture comprises a vision encoder, a visual prompt encoder, and a Large Language Model (LLM). We leverage pre-trained MLLMs such as SPHINX[18, 32] as the base model, chosen for their robust vision-language understanding. To align vision and language features at the pixel level more effectively, we propose a two-stage training strategy: $(S1)$ pre-training for image-visual prompt-text alignment and $(S2)$ supervised fine-tuning following instructions. Furthermore, unlike other methods, we integrate continuous visual prompts (like strokes and scribbles) with bounding boxes during inference, avoiding separate modeling. To effectively adapt the model to this approach, we introduce a noise-based training augmentation strategy. This involves adding noise to sparse visual prompts to simulate the region area for free-form shaped inputs. With these methods in place, users can interact with SPHINX-V using their native language, as well as referring actions such as clicking and drawing, to obtain desired answers about the region of interest as shown in Fig.2.

Figure 2: (a) SPHINX-V has achieved unprecedented performance in text recognition, surpassing other visual prompting MLLMs. Meanwhile, it excels in understanding implicit relationships and conducting complex reasoning. (b) SPHINX-V demonstrates the capability to perceive objects at a pixel level granularity, such as ’a girl holding a pen’. Furthermore, it can perform detailed captioning and inter-relationship analysis on arbitrarily shaped Regions Of Interest.

To further achieve fine-grained pixel-level understanding, high-quality instruction-following training datasets are essential. Currently, several works [70, 67, 74] have developed datasets focused on region-text referring instructions. However, these datasets are mainly limited to natural images, which restricts the model’s applicability after training. Additionally, their scope is often narrow, typically covering only regional descriptions and simple Q&A tasks. To overcome these limitations, we introduce the Multi-domain Visual-Prompting Instruction Dataset (MDVP-Data). Unlike previous datasets, MDVP-Data covers a wider range of image types and interactions, providing a more comprehensive solution. The dataset includes approximately 0.9 million images and 1.6 million query-answer pairs, primarily featuring image-point-text and image-region-text pairings. We assembled this dataset by enhancing existing datasets with segmentation masks or bounding box annotations and by utilizing the capabilities of GPT-4V for data generation. MDVP-Data includes detailed attributes for objects identified by visual prompts, their relationships with nearby entities, and the context within the background. By covering a broad spectrum of pixel-level understanding across diverse image domains, MDVP-Data significantly improves MLLMs’ adaptability and accuracy in various visual scenarios.

To evaluate the effectiveness and robustness of visual prompting models, we introduce MDVP-Bench, a benchmark tailored for assessing models’ pixel-level comprehension. MDVP-Bench encompasses a variety of tasks, including point-level and region-level captioning, which test the models’ ability to describe specific visual elements; inter-relationship analysis, assessing the understanding of connections between entities; and complex reasoning, challenging the models’ ability to infer and deduce from visual information. We anticipate that MDVP-Bench will lay a solid foundation for future research in the realm of visual prompting and multimodal models.

In particular, our paper makes the following contributions:

• We introduce SPHINX-V, a new multimodal large language model designed for visual prompting, equipped with a novel visual prompt encoder and a two-stage training strategy. SPHINX-V supports multiple visual prompts simultaneously across various types, significantly enhancing user flexibility and achieve a fine-grained and open-world understanding of visual prompts.

• We construct MDVP-Data, a comprehensive dataset for multi-domain visual-prompt instruction tuning. This dataset encompasses data for both point-level and region-level understanding, designed to enhance a model’s comprehension ability and robustness.

• We introduce MDVP-Bench, a challenging benchmark designed to evaluate tasks that require a combination of detailed description referrals, inter-relationship analysis, and complex reasoning.

• In a broad range of pixel-level understanding tasks across established benchmarks, our experimental results reveal that our model consistently outperforms other visual prompting models, showcasing SPHINX-V’s exceptional pixel-level understanding.

2 Related Works

2.1 Multimodal large language models (MLLMs)

In recent years, the development of Large Language Models (LLMs) has marked a significant milestone in the field of Natural Language Processing (NLP). LLMs such as the GPT series [49], PaLM [12], and LLaMA [58] have not only achieved remarkable results in text processing but also laid the foundation for the development of multimodal learning. Following this, the emergence of Multimodal Large Language Models (MLLMs), including BLIP-2 [29], Flamingo [1], and PaLM-E [14], further expanded the application scope of LLMs. Moreover, ongoing research into LLMs has explored extending their capabilities beyond text to other modalities. Notably, a fine-grained understanding of vision has emerged as a new focus. Works such as VisionLLM [62] employ language-guided tokenizers to extract vision features at specific granularities to obtain relevant visual features, showcasing the potential in this direction.

2.2 Visual and Multimodal Prompting

Visual and multimodal prompting in deep learning [24, 70, 67, 3, 74] is an emerging area of study. Techniques using visual prompts (e.g., boxes, masks) aim to enhance model performance on specific visual tasks. Key developments include SAM [24] and its enhanced versions, which support a broad range of prompts. However, SAM initially struggled to generate semantic labels vital for complex scene analysis. This gap led to innovations like SEEM [80], HIPIE [63], and Semantic SAM [27] to improve semantic prediction. However, for real-world applications, models require multidimensional semantic analysis to fully grasp and reason about visual scenes.

Recent studies such as GPT4RoI [73], Kosmos-2 [46], Shikra [7], Ferret [67], and GLaMM [51] have enabled MLLMs to achieve region-based image understanding. Colorful Prompting Tuning (CPT)[66] and RedCircle[55] pioneer the use of color overlays and targeted visual cues to enhance models’ interpretative abilities. ViP-LLaVA [3] also allows users to intuitively mark images and interact with the model using natural prompts. Osprey [70] incorporates fine-grained mask regions into language instruction, achieving precise pixel-wise visual understanding. However, this mask-based technology relies on pre-attached segmentation models or externally provided ground truth masks, which limits its flexibility and scope of application. Furthermore, most existing methods do not support referring to multiple targets simultaneously, which significantly limits the model’s ability to perform more complex understanding and reasoning tasks.

Figure 3: Illustrative Example from the MDVP Dataset: This diagram displays the distribution of images drawn from various domains, including nature scenes, texts in the wild, web content, mobile interfaces, documents, and multi-pane graphics. It also features a sample from the GPT-assisted MDVP dataset, emphasizing the diversity and richness of its point-based and region-based instruction-following data.

3 MDVP: Multi-domain Visual-Prompt Instruction Data Generation

In this section, we introduce MDVP-Data, an instruction dataset designed to foster fine-grained pixel-level and open-world image understanding in MLLMs, encompassing approximately 1.6 million multimodal dialogues. MDVP-Data integrates both point-level and region-level instruction data derived from public datasets. It consists of two types of data: $(1)$ a restructured public grounding dataset formatted for visual prompt-based instruction following (Sec. 3.1); and $(2)$ high-quality training pairs developed using meticulously crafted prompt templates, produced through the GPT-4V model (Sec. 3.2). Additionally, we present MDVP-Bench, a challenging benchmark suite designed to evaluate comprehensive pixel-level understanding capabilities under various visual prompts.(Sec. 3.3)

3.1 Inverse-Grounding Visual Prompt-based Dataset

We observe that the grounding capability, which encompasses the ability to localize visual content and link each entity referred to by a noun phrase in the caption to a specific region in the image, essentially represents the inverse process of visual prompt proficiency. Building on this insight, we reorganize existing datasets for Referring Expression Comprehension (REC), Phrase Grounding, and Grounding QA to construct our dataset for visual prompting.

For captioning tasks, we have inverted the inputs and outputs of existing REC and Phrase Grounding datasets, such as Flickr30K [48] RefCOCO/+[23, 69] GCG[51], and GRIT [46], treating the ground truth boxes as input visual prompts, with the corresponding natural language descriptions serving as the answers. For visual prompt-based conversations, we utilize grounding QA pairs from point-QA [40], Visual 7W [79] and VCR [72] datasets. The bounding boxes mentioned in the questions are treated as visual prompt inputs, with the positions of bounding boxes in both questions and answers being substituted with <Mark i> or <Region i>.

3.2 GPT4V-assisted Visual Instruction Data Generation

To construct a more comprehensive and diverse instruction-following dataset, we have collected various multi-domain images along with their unique annotations from publicly available datasets. These encompass natural images, Optical Character Recognition (OCR) in the wild, documents, webpage screenshots, mobile screen captures, and multi-panel images. For each distinct image domain, we have meticulously crafted prompts to facilitate the GPT-4V’s adoption of various roles in the generation of instructional data. Specifically, we task GPT-4V with four distinct assignments: brief captioning, detailed captioning, inter-relationship analysis, and the generation of complex reasoning. Uniquely, the captions created for each image type possess their own distinctive characteristics. For instance, in mobile screen captures, objects can be identified not only as relevant components—icons, text, search bars, URLs, etc.—but information about potential actions is also provided. This includes whether a component is clickable and the expected outcome upon interaction. This diversity significantly enhances the richness and presents challenges for our dataset, covering a broader and more comprehensive range of knowledge domains.

Furthermore, as shown in Fig. 3, to enable GPT-4V to better recognize the objects referred to by visual prompts, we have employed Set-of-Marks (SoM) [65] prompting. This method involves directly marking the highlighted objects on the input images, ensuring that the data created is strongly associated with the referenced objects. Concurrently, we propose incorporating the category information of each object within the prompts, which significantly enhances the quality of the generated data. (For more details on the specific prompts and the methods used for marking the objects, please refer to the Appendix. 0.C.2.)

3.3 MDVP-Bench

To evaluate the proficiency of the MLLM in complex pixel-level image understanding tasks and its versatility across various domains, we initially curated a subset of our MDVP-Data. This subset was subjected to a thorough manual content review and filtering process, culminating in the creation of MDVP-Bench. MDVP-Bench is a comprehensive and challenging benchmark encompassing a wide array of tasks, including concise descriptions, elaborate narratives, analyses of interconnections among different regions, and complex reasoning.

Existing evaluations typically follow the approach of LLaVA [35], utilizing GPT-4 for open-ended evaluation and employing textual descriptions to depict image content. These textual descriptions, often based on image annotations, can lead to scenarios where GPT-4 may not recognize objects or backgrounds not annotated in the scene. Additionally, inherent domain gaps between textual descriptions and images can lead to GPT-4 misunderstanding the image content. To address these limitations, we employ GPT-4V for evaluation in MDVP-Bench. Following the pipeline described in Sec.3.2, we directly annotate images with SoM prompting, submitting both the image and textual questions to GPT-4V for scoring. This approach helps to circumvent potential misinterpretations of image content by GPT-4V. The scoring adheres to the LLaVA-bench guidelines, with scores ranging from 1 to 10, where higher scores indicate superior model performance.

4 Model Architecture

Figure 4: Overview of the SPHINX-V model architecture. (a & b) The proposed two-stage training strategy. (c) Overall model architecture.

In this section, we present SPHINX-V, a model designed for pixel-level referring understanding. As illustrated in Fig. 4(c), SPHINX-V comprises three components, : a mixed vision encoder, a versatile visual prompt encoder, and a large language model (LLM). Given an image alongside coordinates of specific points or regions and instruction language, we perform tokenization and encoding to derive three distinct embeddings: image embeddings $Z_{i}$, visual prompt embeddings $Z_{v}$, and text embeddings $Z_{t}$. Subsequently, we employ two projection matrixs to convert $Z_{i}$ and $Z_{v}$ into language embedding tokens, aligning them with the word embedding space in the language model. Finally, we integrate the image, visual prompt, and language representations, forwarding them collectively to the LLM to attain pixel/region-level understanding capability.

Vision Encoder. Driven by the scalability and effective pretraining of MLLMs, we incorporate the Mixed Vision Encoder (MVE) from the pretrained SPHINX [18, 32]. This encoder generates the image embeddings $Z_{i}=MVE(X_{i})$, operating at an image resolution of $448\times 448$. It expands the scope of image embeddings to encompass multiple scales and high-resolution sub-images, facilitating the efficient encoding of high-resolution images.

Visual Prompt Encoder. In SPHINX-V, we mainly consider two types of visual prompts: points and bounding boxes. As shown in Fig. 4(c), we introduce a new visual prompt encoder. Inspired by SAM [24], initially, we utilizes positional encodings [57] for points and boxes, then adds learnable embeddings for each prompt type, and finally processes them through a linear layer to obtain unified output embeddings. Additionally, our visual prompt encoder accommodates a dynamic number of visual prompts as input. Specifically, we first set a fixed number of input tokens for visual prompts (e.g., 10). Then, based on the validity of the actual input tokens, we equip both valid and invalid tokens with a set of learnable vectors to assist the model in discerning their effective features. (For more details on the visual prompt encoder, please refer to the Appendix. 0.A.)

Large Lanuage Model. We consider LLaMA 2-13B [58] as our large language model (LLM), a transformer-based decoder-only LLM.

Table 1: Statistics of Data in Stage 1.

Type	Raw Data	#Samples
Natural	COCO [31]/LVIS [19] Visual Genome [25] OpenImages [26] Obj365 [54]/V3Det [60] ADE20k [76] Cityscapes [13] Pascal VOC [41] Flickr30k [48]	4.6M
Document layout	Docbank [30] PublayNet [75] DoclayNet [47]/M6Doc [9]	3.1M
OCR Spoting	icdar13 [22]/icdar15 [21] ctw1500 [71]/mlt2017 [43] mlt2019 [42]/totaltext [11] CurvedSynText150k [36]	550K
Android & Web	AITW [52]/SeeClick [10]	300K

Table 2: Statistics of Data in Stage 2.

Task	Raw Data	#Samples
Category Identification	Subset in Stage-1	2.0M
Brief Caption	RefCOCO [23, 69] RefCOCO+ [23, 69] RefCOCOg [23, 69] Visual Genome [25] MDVP-Data	1.2M
Detailed Caption	MDVP-Data	752K
Relationship Analysis	Visual Genome [25] MDVP-Data	1.1M
Inverse-Grounding Description	GRIT [46] Flicker30K [48] OpenPsgGCG [51]	1.2M
Whole Image Caption	shareGPT4v [8] laionGPT4v [5]	1.2M
Q&A and Resoning	VCR [25]/Visual7W [79] Osprey-724K [70] MDVP-Data	2.3M

Training Strategy. The training process of our SPHINX-V model consists of two important stages.

$\bullet$ Stage 1: Image-Visual Prompt-Text Alignment Pre-training. By employing a visual prompt encoder, we initially freeze both the pre-trained vision encoder and the pre-trained LLM. Then, we focus on training the features of visual prompts to align with the features of image-visual prompt-text. Following the approach in LLaVA [35], we implement an MLP as the connector between visual prompts and language to enhance the multimodal prompting capabilities of the model. At the first stage, we gather and reconstruct open-source datasets, including those for object detection, semantic segmentation, layout detection, spotting, and more, to build our training data. Specifically, for point inputs, we initially randomly sample from semantic segmentation images, where each point has a corresponding pixel-level label annotation. Similarly, for instance segmentation, we can sample points from the mask of each object, with these points corresponding to the label annotation of that object. For box inputs, we directly utilize the ground truth bounding boxes provided by detection datasets as visual prompt inputs, allowing the model to recognize their corresponding labels. The structure of our stage-1 pre-training data is as follows:

As illustrated in Table 2, we have gathered a diverse array of detection and segmentation datasets covering multiple image domains. These datasets encompass a wide range of label types, including elements of the natural world (persons, animals, stuff, etc.), document elements (titles, paragraphs, images, tables, etc.), OCR (text recognition), and screenshots (icons, text, search bars, etc.). With such rich and extensive data for pre-training, SPHINX-V is equipped to develop capabilities for visual prompting and preliminary category identification.

Furthermore, to support free-form shape visual prompts, we propose a noise-based training augmentation in the pre-training stage to simulate the region area required when inputting free-form shaped visual prompts. Specifically, for box-type visual prompts, we introduce a certain degree of noise based on the size of the box itself. This noise may cause the ground truth bounding box to be slightly larger than the target object or only cover part of the target object, thereby approximating a free-form outer rectangle. For points, we sample several pixels within the mask of a target object. These pixels are directed towards the same target object during training, thus achieving precise point-based referring.

$\bullet$ Stage 2: Multi-Task Supervised Finetuning. At the second stage, we load the weights trained during stage 1 and keep the vision encoder and visual prompt encoder weights frozen. We then fine-tune the visual prompt projector and the LLM model. Our focus is on enhancing SPHINX-V’s ability to accurately interpret user instructions and handle diverse pixel-level understanding tasks, such as detailed captioning, inter-relationship analysis, and complex reasoning. Table 2 outlines all the data utilized during stage 2 fine-tuning, which includes our proposed MDVP dataset, Visual Genome (VG)[25], Visual Commonsense Reasoning (VCR)[72], Visual7w [79], and Osprey-724k [70]. Furthermore, we reformat the grounding dataset for compatibility with visual prompt-based instructions. Lastly, we ensure the model retains the capability to provide detailed captions for the entire image when given a visual prompt that encompasses most of the image’s content. After the instruction fine-tuning in stage 2, SPHINX-V becomes proficient in comprehending complex scenarios based on user instructions and specific regions of interest.

Inference. Although SPHINX-V is limited to modeling point-level and box-level features of visual prompts, the noise-based training augmentation proposed in the stage 1 enables users to freely draw on the image, greatly increasing the flexibility of user usage. Specifically, by pre-processing these free-form visual prompts, we can obtain their corresponding bounding boxes, and then feed these box coordinates into SPHINX-V for inference.

5 Experiments

5.1 Implementation Details

We utilized AdamW [38] as our optimizer and employed flash attention [16] to improve computational efficiency. During the first stage of training, we set the initial learning rate to $4e-5$. For the second stage, the learning rate was adjusted to $1e-5$. The input images were processed at a resolution of 448 x 448 pixels, and the maximum sequence length for the Large Language Model (LLM) was established at 2048. The entire training procedure was conducted on 8 NVIDIA A100 GPUs, each equipped with 80GB of memory. For all evaluation experiments, we opted for a zero-shot testing approach, refraining from further fine-tuning on specific datasets.

5.2 Referring Object Classification

This task is defined as follows: the question targets a specific area within the image, requiring the model to identify the object within that designated region. Following [70], we employ two semantic relevance indicators—Semantic Similarity (SS) and Semantic Intersection over Union (S-IOU)[53]—to assess a model’s classification prowess on the validation sets of object-level LVIS [19] and part-level PACO [50] datasets. As the results shown in Table 3 indicate, SPHINX-V achieves 83.16% in SS and 58.64% in S-IoU based on point-level visual prompts, and 87.06% in SS and 62.90% in S-IoU based on box-level visual prompts on the LVIS dataset. Both sets of results significantly outperform the state-of-the-art method by a notable margin. Furthermore, SPHINX-V surpasses the previous best, Osprey, by 6.82% in SS and 2.32% in S-IoU on the PACO dataset, demonstrating its robust capability in fine-grained part-level classification.

We also conducted tests on traditional closed-set object classification tasks. To ensure the model outputs names of categories within the closed set, we followed the approach described in Ferret [67] and adapted our evaluation to a binary-choice format. We posed questions such as ’Please identify the label of the marked region in the image. Is it (Class A) or (Class B)?’. The results, as shown in Table 4, further demonstrate SPHINX-V’s excellent classification ability.

Table 3: The results of referring object classification on LVIS and PACO.

Method	LVIS		PACO
	Semantic Similarity	Semantic IoU	Semantic Similarity	Semantic IoU
LLaVA-1.5 [34]	48.95	19.81	42.20	14.56
Kosmos-2 [46]	38.95	8.67	32.09	4.79
Shikra-7B [7]	49.65	19.82	43.64	11.42
GPT4RoI-7B [73]	51.32	11.99	48.04	12.08
Ferret-7B [67]	63.78	36.57	58.68	25.96
Osprey-7B [70]	65.24	38.19	73.06	52.72
SPHINX-V(Point)	83.16	58.64	76.18	51.13
SPHINX-V(Box)	87.06	62.90	79.88	55.04

Table 4: Results of referring object classification and optical character recognition (OCR) on COCO text. We randomly perturb the center positions and scale the dimensions of box visual prompts to simulate free-form prompts.

Method	LVIS			OCR (COCO Text)
	Point	Box	Free-Form	Box
LLaVA [34]	50.1	50.3	-	-
Kosmos-2 [46]	-	60.3	-	-
Shikra-7B [7]	57.8	67.7	-	-
GPT4RoI-7B [73]	-	61.8	-	-
Ferret-7B [67]	67.9	79.4	69.8	-
Ferret-13B [70]	68.4	80.5	71.0	-
ChatSpot-7B	-	64.5	-	31.8
SPHINX-V(Ours)	86.46	89.82	88.96	45.44

5.3 Regional Optical Character Recognition.

Optical character recognition (OCR) is a task focused on identifying text in images, serving as a fundamental aspect of visual entity recognition. Following [74], we utilize the COCO-Text dataset [59] to gauge the regional text recognition prowess of SPHINX-V. Employing the ground-truth bounding boxes supplied within the dataset, we prompt the model with requests such as "Please provide the ocr results of each marked region in the image." SPHINX-V then analyzes and response the textual content present within the specified regions. Given that most visual prompt-based models do not support OCR, we compare our results with ChatSpot under identical zero-shot settings. As illustrated in Table 4, our model surpasses the ChatSpot by 13.64%, showcasing the promising regional capability of text recognition.

Moreover, as shown in the Fig. 5, SPHINX-V is capable of not only recognizing text in the referring region but also possesses enhanced understanding and descriptive abilities. For instance, it can further describe characteristics such as the font type, color, and background of the text, and understand the role of the text within the image area. This fully demonstrates SPHINX-V’s exceptional pixel-level understanding capabilities.

5.4 Region-Level Captioning

Brief Region Description. We provide quantitative comparisons for the region-level captioning task using both mask- and box-based approaches [46, 51, 15, 70]. Specifically, we employ a box visual prompt and a text prompt, such as "Please provide a brief description of each marked region in the image," to prompt SPHINX-V to concisely describe the content of the targeted region. Experiments were conducted on the validation sets of RefCOCOg. The comparison results, shown in Table 6, demonstrate promising outcomes when compared to other methods. (More experimental results can be found in Appendix. 0.B.1)

Table 5: Detailed region-level captioning performance on the RefCOCOg validation set.

Method	Detailed Description
LLaVA-1.5 [34]	71.11
Kosmos-2 [46]	40.89
Shikra-7B [7]	40.97
GPT4RoI-7B [73]	49.97
Osprey-7B [70]	77.54
SPHINX-V(Ours)	92.19

Table 6: Brief region captioning performance evaluated on the validation set of RefCOCOg.

Method	Type	METEOR	CIDEr
GRIT [67]	Box	15.2	71.6
Kosmos-2 [46]	Box	14.1	62.3
GLaMM [51]	Box	16.2	105.0
Osprey-7B [70]	Mask	16.6	108.3
SPHINX-V(Ours)	Box	23.9	162.5

Detailed Region Description. To further evaluate the detailed area description capabilities of our model and other visual prompting MLLMs, inspired by [70, 34], we leverage GPT-4 to comprehensively measure the quality of the responses generated by the model to the input referring regions. Specifically, in the same manner as Osprey [70], we sample 80 images from the validation set of RefCOCOs [23, 69] for generating detailed region caption, using box visual prompt and text prompt like "Please provide a detailed description of each marked region in the image.". Subsequently, GPT-4 was utilized to assess the detailed captions generated by MLLMs. The evaluation scores span from 1 to 10, with higher scores reflecting enhanced performance. To evaluate the efficacy of MLLMs, we determined the ratio of the predicted answer score to that of GPT-4, expressing this ratio as a percentage. The results shown in Table 6 indicates that SPHINX-V achieves the best performance with 92.19% accuracy, significantly outperforming other region-based and mask-based methods.

Table 7: Results on LLaVA-Bench and Ferret-Bench.

	LLaVA-Bench			Ferret-Bench
	Conversation	Detail Description	Complex Reasoning	Referring Description	Referring Reasoning
LLaVA [35]	85.4	68.3	92.1	41.4	31.7
Kosmos-2 [46]	71.7	63.4	74.9	51.8	33.7
Shikra-7B [7]	80.6	70.7	88.1	46.0	41.6
Ferret-7B [67]	84.4	79.4	96.3	68.7	67.3
Ferret-13B [67]	85.2	80.9	96.4	70.6	68.7
SPHINX-V(Ours)	85.0	86.4	91.3	76.4	70.4

Table 8: Results on different domains of MDVP-bench. *Osprey accepts the ground truth mask of the referring object as visual prompt input.

Domain	Natural		OCR		Multi-panel		Screenshot
	Box	Point	Box	Point	Box	Point	Box	Point
Osprey* [70]	86.40	-	18.33	-	48.56	-	28.84	-
Ferret-7B [67]	84.92	81.42	21.24	12.20	44.42	32.08	26.40	10.62
Ferret-13B [67]	86.67	83.40	27.68	12.20	49.83	37.65	30.92	14.40
SPHINX-V(Ours)	88.82	92.95	78.49	76.29	77.30	79.08	80.07	54.05

Figure 5: Visual examples of SPHINX-V.

5.5 Comprehensive Assessment: LLaVA-Bench, Ferret-Bench and MDVP-Bench

We use LLaVA-Bench to compare SPHINX-V with previous models [34, 7, 46, 70] to evaluate whole image-level understanding capability. Given that the questions in LLaVA-Bench are based on the whole image, we employ bounding boxes of nearly full-image size as visual prompts. Furthermore, we use Ferret-Bench and our proposed MDVP-Bench to evaluate pixel-level understanding capability. Experiments were conducted on the Referring Description and Referring Reasoning tasks within Ferret-Bench.

As the results presented in Table 7, since we did not incorporate open-source image-level VQA data into our training dataset, it is observable that in LLaVA-Bench, our model performs slightly below the current state-of-the-art methods in conversation and complex reasoning evaluations. However, it still significantly leads in the capability of detailed captioning. Moreover, in the Ferret-Bench assessment focused on pixel-level image understanding, whether in referring description or referring reasoning, our model achieved the best performance. This demonstrates SPHINX-V’s powerful fine-grained part-level understanding and reasoning abilities.

For MDVP-Bench, we mixed different tasks together to obtain a comprehensive assessment of the model’s overall capabilities, applying evaluation process mentioned in Sec 3.3. As the results shown in Table 8, compared to Osprey and Ferret, our SPHINX-V significantly leads in evaluations across multiple domains. In addition, it can be observed that the scores for box prompts are sometimes lower than those for point prompts. Upon reviewing the responses to each question, we discovered that for tasks with fewer visual prompts on an image, box-based prompts perform better, while in tasks involving more visual prompts on an image, simple point prompts demonstrate significantly better performance. This finding reveals the modeling function of our visual prompt encoder and their capability to model different visual prompts, inspiring future research and improvement of their functionality.

Figure 6: Comparative visual examples from various methods.

5.6 Visualization

We provide visual examples to further illustrate the pixel-level spatial understanding capabilities of our SPHINX-V across a diverse range of visual prompts. Fig. 5 displays a variety of visual scenarios from different domains, such as natural scenes, mobile and web interfaces, optical character recognition (OCR) in natural settings, document layouts, and multi-panel images. In these complex situations, SPHINX-V showcases its proficiency in accurately responding to queries, highlighting its robust and versatile capabilities. (More visualizations could be found in Appendix. 9)

Furthermore, Fig.6 provides comparisons with existing image-level and region-level methods. Our methodology yields superior scene comprehension results, offering a nuanced and detailed analysis when compared to existing image-level and region-level approaches.

5.7 Ablation Study

To evaluate the effectiveness of the key elements of our design, we conduct the following ablation experiments. Given the extensive amount of training data, our comparison was limited to the first 50k training iterations.

Table 9: Comparisons across different modeling format of visual prompts and training strategy.

Method	LVIS		PACO
	Semantic Similarity	Semantic IoU	Semantic Similarity	Semantic IoU
Alpha Blending	45.5	21.7	39.6	14.2
Text	47.3	25.5	43.47	18.3
Encoder(Ours)	55.9	30.1	52.3	22.4
One-Stage	43.1	19.2	39.2	12.5
Two-Stage(Ours)	55.9	30.1	52.3	22.4

The Effectiveness of Visual Prompt Encoder. To assess the effectiveness of our visual prompt encoder, we explored two alternative methods for incorporating visual prompts into MLLMs: (1) overlaying the visual prompts on the original image via alpha blending with an alpha value of 0.5; and (2) explicitly including the coordinates of the visual prompt in the instructional text prompt. As shown in Table 9, while these alternative methods enabled the MLLM to successfully identify the regions indicated by the visual prompts, their performance was not as strong as that achieved with our visual prompt encoder. This suggests that our visual prompt encoder possesses superior capabilities for processing visual prompts.

The Two-stage Training Strategy. To validate the effectiveness of our proposed two-stage training strategy, we conducted experiments in which we bypassed the pre-training phase (stage 1) and directly set both the visual prompt encoder and the LLM to a trainable state for model training. As depicted in Table 9, the results clearly demonstrate that, within the same training duration and using identical data, the two-stage training approach significantly outperforms the single-stage method. This improvement is due to the two-stage training strategy’s notable acceleration of the model’s convergence speed, thereby underscoring the effectiveness and necessity of the two-stage training paradigm.

6 Conclusion

In summary, we introduce a new multimodal large language model for visual prompting named SPHINX-V, which exhibits promising capabilities in pixel-level image understanding. With our proposed visual prompt encoder and two-stage training strategy, SPHINX-V is able to support various types of visual prompts, including points, boxes, and free-form shapes, significantly enhancing user flexibility. Additionally, we have curated the MDVP dataset, comprising 1.6M high-quality image-point-text and image-region-text pairs for model training, and the MDVP benchmark, which is both comprehensive and challenging, for evaluation purposes. Our superior performance in established visual prompting tasks—including referring object classification, region-level captioning, regional OCR, and referring reasoning—underscores the efficacy and robustness of SPHINX-V. We anticipate that our contributions will lay a solid foundation for further exploration in the field of intelligent visual interaction systems.

Appendix

Appendix 0.A More Details of Architecture

Image Encoder. In general, the image encoder can be any network that outputs a $C\times H\times W$ image embedding. Motivated by scalability and access to strong pre-training, we employ a Mixture of Visual Experts (MoV) with mixed scales and high-resolution sub-images to process high-resolution inputs [18, 32]. Specifically, we use two complementary vision encoders: DINOv2 [44] and CLIP-ConvNeXt [37], to capture a diverse visual representation. During training, all images are by default scaled and zero-padded to a high resolution of $448\times 448$, and then divided into four $224\times 224$ sub-images. For images with large aspect ratios, such as $2:1$, this results in fully-padded sub-images that are filled entirely with zero-value pixels. To address this issue, we follow a strategy [18] that involves using a learnable skip token to replace the fully-padded sub-image. This provides explicit relative positional information, enabling Language Learning Models (LLMs) to identify the positions between useful sub-images more effectively.

Visual Prompt Encoder. In Fig. 7, we present our approach for the integration of multiple visual prompts (VPs) into the visual prompt encoder. Specifically, for point-type inputs, we represent VPs as the sum of a positional encoding [57] of the point’s coordinates. For box-type inputs, inspired by [24], VPs are conceptualized as the aggregate of positional encodings for the coordinates of both the ’top-left corner’ and the ’bottom-right corner’. We then proceed to dynamically assess the efficacy of the feature set associated with each VP, applying one of two learned embeddings to indicate the VP’s validity. To further delineate and model the positions of the center point, the ’top-left corner’, and the ’bottom-right corner’, we allocate three distinct learned embeddings for each. The culmination of this process involves the incorporation of an MLP as the output layer, tasked with generating the final embeddings of the visual prompts with a dimensionality of 512.

Figure 7: Details of the visual prompt encoder.

Appendix 0.B More Experiments

0.B.1 Brief Region Description

To further evaluate the captioning capability of SPHINX-V, we conducted tests on the evaluation set of Visual Genome [25]. As the comparison results shown in Tab. 10, our SPHINX-V model exhibits promising performance compared to other models that utilize various types of prompts. It achieves a METEOR score of 24.3 and a CIDEr score of 197.3, significantly outperforming the state-of-the-art TAP(ViT-L) [45] approach by 6.8 and 46.6, respectively. These results highlight the superior capability of our model in generating semantically relevant descriptions for object regions.

Table 10: Comparison of various methods on the Visual Genome dataset.

Method	Prompt Type	Visual Genome
		METEOR	CIDEr
GRiT [64]	Bounding Box	17.1	142.0
GPT4ROI-7B [73]	Bounding Box	17.4	145.2
GPT4ROI-13B [73]	Bounding Box	17.6	146.8
ASM [61]	Bounding Box	18.0	145.1
SCA(GPT2-774M) [20]	(8 Caption + 6 Task) Tokens	17.4	148.8
SCA(LLama-3B) [20]	(8 Caption + 6 Task) Tokens	17.4	149.8
TAP(ViT-B) [45]	1 Semantic Token	17.4	149.2
TAP(ViT-L) [45]	1 Semantic Token	17.5	150.7
SPHINX-V(ours)	Bounding Box	24.3	197.3

Table 11: Validation Accuracy on VCR dataset.

Model	Q $\rightarrow$ A (%)	QA $\rightarrow$ R (%)	Q $\rightarrow$ AR (%)
ViLBERT  [39]	72.4	74.5	54.0
Unicoder-VL  [28]	72.6	74.5	54.5
VLBERT-L  [56]	75.5	77.9	58.9
ERNIE-ViL-L  [68]	78.52	83.37	65.81
VILLA-L  [17]	78.45	82.57	65.18
GPT4RoI-7B [73]	87.4	89.6	78.6
ViP-LLaVA-Base-7B [3]	87.66	89.80	78.93
SPHINX-V(Ours)	89.65	90.23	89.63

0.B.2 Region-Level Reasoning

To evaluate the reasoning capabilities of SPHINX-V, we utilized the Visual Commonsense Reasoning (VCR) dataset [72], a challenging benchmark designed to assess a model’s high-level cognitive and commonsense reasoning abilities within context. The VCR dataset comprises multiple-choice questions that require an understanding of the scene depicted in an image. Each question (Q) is accompanied by four possible answers (A), necessitating the model to not only identify the correct answer but also provide a rationale (R) supporting its selection. This process underscores the model’s proficiency in interpreting and justifying visual elements within specific contexts, with accuracy serving as the evaluation metric. Following the evaluation approach employed in LLaVA-ViP [34] and GPT4RoI [73], we fine-tuned our SPHINX-V model using the training set of VCR. As shown in the comparison results in Tab. 11, our model achieves the highest performance scores of 89.65%, 90.23%, and 89.63% across three distinct evaluation methods, respectively, showcasing its proficiency in visual commonsense reasoning.

Appendix 0.C More Details of Dataset

In this section, we outline our methodology for reconstructing public grounding datasets and our application of GPT-4V to produce instruction-based data spanning a variety of domains, leading to the development of the MDVP Dataset. Fig.8 depicts an example from our Inverse-Grounding dataset. Sec.0.C.2 showcase the prompts used to generate data across these diverse domains with GPT-4V.

0.C.1 Examples of Inverse Grounding

Using two sample images from the Flickr30k dataset [48] as illustrations, the phrase grounding task is defined as identifying the bounding boxes in an image that correspond to various referring phrases, based on a given ground truth (gt) sentence. In a twist on this task, we invert the annotation process: the model, provided with bounding boxes, is tasked with generating the corresponding gt sentence. This process is depicted in Fig. 8.

Figure 8: Examples of Inverse-Grounding data.

0.C.2 Prompts for Data Generation

We carefully designed prompts to facilitate data generation by GPT-4V, assigning it various roles and embedding category-specific information within these prompts. Table 12 outlines a universal prompt template applied across different domains. Meanwhile, Tables 13, 15, 15, 17, and 17 detail prompts tailored to specific domains. An illustrative example of GPT-4V’s response during data generation is provided in Table 18.

Table 12: The public prompt template used to feed to GPT-4V for data generation. Public prompts for different domains exhibit slight variations. For instance, in the screen-shot domain, "In the image" is modified to "In the screen-shot". The highlighted <Role> and <Format> in red are domain-specific.

Table 13: The <Role> and <Format> prompt template used for natural images domain.

Table 14: The <Role> prompt template used for screenshots domain. The <Format> can be referenced from natural images domain in Tab.13.

Table 15: The <Role> prompt template used for document domain. The <Format> can be referenced from natural images domain in Tab.13.

Table 16: The <Role> prompt template used for OCR-spotting domain. The <Format> can be referenced from natural images domain in Tab.13.

Table 17: The <Role> prompt template used for multi-panels images domain. The <Format> can be referenced from natural images domain in Tab.13.

Table 18: An example response from GPT-4V during data generation.

Figure 9: Additional visual examples of SPHINX-V utilizing arbitrary-shaped visual prompts across multiple domains.