Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs

Prism is characterized by a modular design that decomposes the process of solving visual questions into two stages, consisting of a perception module and a reasoning module, both of which can be flexibly replaced, as depicted in Fig. 1. The perception module, typically a VLM, initially follows the instruction to extract visual information from images and articulates this information in textual form. The instruction can be generic or query specific, ie., written by the reasoning module given the question (text-only) as contextual information. Meanwhile, the reasoning module, usually an LLM, performs text-based reasoning on the textual information to generate answers to the questions. To assess the perception capabilities of various VLMs, we carefully select an appropriate benchmark corresponding to the principle we illustrate in Sec. 2.2. In Sec. 2.3, we describe how we utilize the Prism framework to assess the perception and reasoning capabilities of VLMs, respectively. To substantiate our belief in the potential of combining a small-scale VLM with a powerful LLM, we train an approximately 2B-parameter VLM based on the LLaVA architecture to serve as a visual captioner and integrate it into Prism as the perceptual module, as detailed in Sec. 2.4.

Numerous multimodal benchmarks [liu2023mmbench, Fu2023MMEAC, li2023seed, lu2023mathvista, yue2023mmmu, chen2024we] exist, each evaluating the capabilities of VLMs from various perspectives. To maximize the utility of the Prism framework, careful selection of the benchmark is imperative for the decoupling analysis. In summary, we adhere to the following principles for benchmark selection: 1) Vision Indispensability: The benchmark must require visual information for question solving, and questions that can be answered without utilizing visual information are excluded; 2) Minimal Data Leakage: The visual questions should not be part of the model’s training data; and 3) Complexity: The solving process of the visual question should involve both perception and reasoning components. Considering these three principals, we select MMStar [chen2024we] as the primary benchmark for the decoupling analysis and ablation. MMStar ensures vision indispensability and makes a concerted effort to minimize data leakage, while many other benchmarks are plagued by the two issues.

Prism functions as an analytical framework for evaluating the perception and reasoning capabilities of a given VLM. In this section, we elaborate on the methodology for evaluating these capabilities.

To assess the perception capabilities of various VLMs, we employ ChatGPT (GPT-3.5-turbo-0125) as the reasoning module and standardize instructions for description within Prism. With the reasoning module and instructions fixed, the final accuracy of the visual questions is solely determined by the quality of the visual information extracted. Under the controlled setting, we consider the VQA accuracy as a proxy to measure the perception capability of VLMs.

Within the Prism framework, the instructions for visual information extraction are crucial as they are designed to elicit the fundamental perceptual capabilities of VLMs. We have adopted two types of instructions to assess the perceptual abilities of models:

1. 1.

   Generic Instruction: a standardized, universal instruction aimed at extracting and describing the basic elements present in an image;

2. 2.

   Query-Specific Instruction: a combination of the generic instruction and an incremental instruction that directs the VLM to provide a detailed account of the visual information relevant to the question. The incremental instruction is crafted by the reasoning module given the text-only question.

Employed with the generic instruction, Prism offers a straightforward decoupling pipeline. Meanwhile, using query-specific instructions is a more realistic setting and can better realize the full potential of the Prism framework. A Prism decoupled GPT-4o (GPT-4o for both perception and reasoning) achieves almost the same quantitative performance compared to the end-to-end GPT-4o. Analyzing the results, we find that with the reasoning module and instruction controlled, VLMs with language models of different sizes display a much narrowed gap in perception performance compared to the results under the end-to-end setting.

Besides evaluating the perception performance, Prism can also roughly measure the reasoning capabilities of VLMs. When used to solve general visual questions in an end-to-end manner, the VLM implicitly performs reasoning with its language encoder. Alternatively, Prism provides an explicit pipeline, in which an off-the-shelf LLM is separately used to predict the answer based on the visual information. For small-scale VLMs (7B-parameter, etc.), we find that Prism equipped with the VLM and ChatGPT-3.5 outperforms the end-to-end VLM in quantitative performance, especially for reasoning related VQA. The results reveal that for small-scale VLMs, the overall performance can be heavily constrained by the parameter size of the language model.

Beyond serving as an evaluation framework, Prism can also function as an efficient vision-language task solver. The perception module in Prism can incorporate one or multiple VLMs to extract high-quality visual information. Concurrently, the reasoning module can be instantiated with a powerful LLM to harness its advanced reasoning capabilities.

In Sec. 2.3, we observed that VLMs paired with language models of varying sizes exhibit similar performance in perception capability. This suggests that it is promising to employ small-scale VLMs to generate informative visual descriptions, serving as an efficient perception module. To validate this concept, we conduct extensive experiments to train visual captioners, utilizing the widely adopted LLaVA architecture and only open-source datasets. Below, we elaborate on the specific settings considered for this ablation study in detail:

Instruction Tuning Data. We use ALLaVA-Caption-4V and Evol-Intruct-GPT4-Turbo-143K in ALLaVA [chen2024allava] as our instruction tuning data. The former comprises 715K pairs of images and detailed visual captions produced by GPT-4v, whereas the latter contains 143K text instruction tuning data, generated by GPT-4-Turbo. Utilizing the descriptive data for instruction tuning better triggers the VLM’s ability to extract and articulate visual information more effectively, compared to instruction tuning data in QA formats [liu2023visual, Liu2022VisualSR, textvqa, masry2022chartqa].

Model Architecture. To investigate the impact of vision encoder in LLaVA, we experimented with multiple encoders, including CLIP ViT-L/14 [radford2021learning], SigLip-SO400M [zhai2023sigmoid], and InternViT-6B [chen2023internvl]. For the language encoder, we tested two lightweight variants of InternLM2 [cai2024internlm2]: InternLM2-7B and InternLM2-1.8B. The inference of all combinations (excluding InternViT-6B) can be efficiently executed on consumer-level GPUs such as RTX 4090, etc. During instruction tuning, we maintain the vision encoder fixed and apply QLoRA [dettmers2023qlora] to the language encoder.

Regarding the vision backbone, SigLip exhibits relatively superior performance. Additionally, we observed that a larger language model only results in minor differences in perception performance. Our findings indicate that, within the Prism framework, a 2B vision captioner can achieve strong perception performance on par with LLaVA-NeXT [liu2024llavanext] equipped with a 34B language backbone.

Evalution Details. We use Prism to evaluate the capabilities of various VLMs, which can be categorized into two major groups: (a) Proprietary VLMs, including GPT-4o (20240513) [OpenAI2023GPT4TR], GPT-4v (20231106) [OpenAI2023GPT4TR], GeminiPro-V [team2023gemini], and Qwen-VL-Max [bai2023qwen]; (b) Open-Source VLMs, including LLaVA-v1.5 [liu2023improved], InternLM-XComposer2 [internlmxcomposer2], mPLUG-Owl2 [ye2023mplugowl2], LLaVA-NeXT [liu2024llavanext], InternVL-Chat-v1.5 [chen2024far], DeepSeek-VL [lu2024deepseek], MiniCPM-V-2 [minicpm2024]. When integrating these VLMs as perception modules within Prism, we employ greedy decoding and limit the maximum number of output tokens to 512. The evaluation encompasses both generic and query-specific instructions. Unless otherwise specified, GPT-3.5-Turbo-0125 is adopted as the reasoning module. All evaluations are conducted using VLMEvalKit [2023opencompass]. Further details on the evaluation are provided in Sec. B.1.

Post-Processing. Within the Prism framework, the LLM (particularly proprietary APIs) in the reasoning module often declines to answer questions due to insufficient clues in the extracted visual information. When Prism is employed as an evaluation framework, we classify this as a failure of perception for the visual question and refrain from any post-processing. For fair comparisons, whether Prism is contrasted with other VLMs as a vision-language task solver or different LLMs are evaluated as reasoning modules with varying rejection rates, a random choice is utilized as a fallback when option matching fails.

We present the primary evaluation results of various VLMs’ perception performance in Tab. 1. Results are categorized under two instruction types: Generic Instruction and Query-Specific Instruction.

Generic Instruction Results. When using generic instructions, GPT-4o exhibits exceptional performance across a variety of tasks. InternVL-Chat-v1.5 achieves the highest overall perception performance among open-source VLMs, nearly on par with the proprietary GPT-4v. LLaVA-NeXT (Yi-34B) demonstrates strong coarse and fine-grained perceptual abilities but lags behind GPT-4v in recognizing abstract elements in LR and ST visual questions. Smaller open-source VLMs, such as mPLUG-Owl2 and LLaVA-v1.5-7B, struggle with both coarse and fine-grained perception and encounter difficulties in recognizing essential elements in reasoning-related VQA.

Query-Specific Instruction Results. GPT-4o demonstrates significantly superior performance compared to all other models in the extraction and expression of visual information across all dimensions. GPT-4v is equally adept at handling coarse perception contents. Among open-source VLMs, InternVL-Chat-v1.5 excels in perception tasks across all dimensions excluding LR. Its performance not only surpasses that of other open-source VLMs but is also slightly ahead of GPT-4v. Occasionally, a decline in performance can be observed for specific VLM-task pairs when using query-specific instructions instead of generic ones, which can stem from difficulty in understanding query-specific instructions for specific domains.

Are Proprietary Models Getting Ahead of the Game? When employing both generic and query-specific instructions, proprietary VLMs, particularly GPT-4o, significantly surpass other models in perceptual capabilities and can adeptly manage a wide range of tasks, as demonstrated in Tab. 1. Certain open-source models, such as InternVL-Chat-v1.5 and LLaVA-NeXT (Yi-34B), have achieved notable performance, approaching the capabilities of proprietary VLMs like GPT-4v and GeminiPro-V. Other open-source models, due to their limited perceptual abilities, generally perform slightly worse in Math and ST assessments. Notably, MiniCPM-V-2, a lightweight VLM with $\sim$3B parameters, displays better perceptual performance compared to some 7B VLMs.

The Gap between Perception Ability and End-to-End Performance. In addition to solving visual questions in an end-to-end manner, Prism provides an alternative pipeline where the VLM is solely utilized for perception. The distinction between these two methods lies in the reasoning process: the former conducts reasoning internally within VLMs, whereas the latter performs reasoning based on VLM extracted information using an external LLM (ChatGPT). The comparison between these two approaches on MMStar is depicted in Fig. 2. For state-of-the-art large-scale VLMs such as GPT-4o and InternVL-Chat-v1.5, which are expected to possess excellent reasoning capabilities, employing an external ChatGPT for reasoning may diminish overall performance. Conversely, for most small-scale VLMs, using ChatGPT for reasoning significantly improves their performance, particularly in reasoning-related VQA, as shown in Fig. 5. This phenomenon indicates that the overall performance of small-scale VLMs can be heavily constrained by the size of the language model. To investigate whether the reasoning ability of ChatGPT constrains state-of-the-art VLMs, we implemented a Prism pipeline that decouples GPT-4o by using it as both the perception and reasoning module. The result reveals that this Prism pipeline, with post-processing, achieves an overall accuracy of 61%, nearly identical to the end-to-end GPT-4o performance of 61.6%.

How does Language Model Size Affect Perception Ability? During evaluation, we observe that the LLaVA-v1.5 series shows no significant improvement when using a larger language model (Vicuna-13B instead of Vicuna-7B, etc.). This suggests that perception performance may be independent of the language model size when using a relatively low-resolution vision backbone. However, the situation appears to differ with LLaVA-NeXT. Quantitative results for the LLaVA-NeXT series tell that scaling up the language model slightly enhances model perception, particularly when using query-specify instructions. Through a detailed qualitative analysis, we identified the primary factors contributing to the superior performance of larger LLaVA-NeXT models over smaller ones as follows: (1) More Elaborate Expression: Models equipped with a larger language encoder exhibit enhanced ability to articulate visual information. More detailed and organized narratives make it easier for the reasoning module to answer the question; (2) More Adaptive to Instruction: Larger language backbones entitle the model with a better understanding of instructions, yielding more suitable textual visual information for reasoning, particularly in response to query-specific instructions. In Fig. 3, we provide some qualitative results about the two typical modes.

We perform one-stage training for two epochs using ZeRO2 with XTuner [2023xtuner] on 8 NVIDIA A800-80GB GPUs, and the training lasts less than a day. The training data include of one copy of ALLaVA-Caption-4V and two copies of Evol-Intruct-GPT4-Turbo-143K. The batch size is set to 128 for PrismCaptioner-2B and 64 for PrismCaptioner-7B. We utilized the AdamW optimizer and 2e-4 learning rate, with the warm-up ratio set to 0.03 and ($\beta_{1},\beta_{2}$) set to (0.9, 0.999). No weight decay was applied and a maximum norm value of 1 is applied for gradient clipping. Full details about QLoRA are presented in Sec. B.2.

We conduct a thorough evaluation of PrismCaptioners across multiple benchmarks, employing GPT-3.5-Turbo-0125 and Llama-3-70B-Instruct as the reasoning module. We utilize MMStar as our primary benchmark to assess comprehensive multimodal capabilities. For domain-specific evaluations, we choose AI2D to gauge diagram comprehension, MMMU for expert knowledge assessment, and MathVista to test mathematical proficiency. The results are presented in Tab. 5. In line with the benchmark selection principles outlined in Sec. 2.2, we apply a consistent filtering strategy across AI2D, MMMU, and MathVista, mirroring that of MMStar, to retain only vision-essential and uncontaminated questions, denoted by the suffix (F). In addition to comparisons with existing open-source and proprietary VLMs, we also assess two baseline models, LLaVA-InternLM2-[1.8B/7B], which are PrismCaptioners trained on LLaVA-v1.5 instruction tuning data. We also compare PrismCaptioners and ShareCaptioner, an open-source VLM designed for generating informative image captions, under identical Prism framework configurations.

As depicted in Tab. 5, each PrismCaptioner, with an external potent LLM for reasoning, markedly surpasses its corresponding end-to-end baseline. PrismCaptioners also outperform ShareCaptioner across all multimodal benchmarks. For the 7B variant, the integration of Llama3 results in a substantial enhancement, positioning PrismCaptioner-7B as a highly competitive vision-language solver, particularly on MMStar and MMMU. For PrismCaptioner-2B, employing ChatGPT yields superior results, outperforming nearly all 7B VLMs in general aptitude, expert knowledge, and mathematical skills. Remarkably, it achieves performance levels on par with some ten times larger VLMs, such as LLaVA-InternLM2-20B, Yi-VL-34B, and Emu2-Chat. This demonstrates that Prism enables the creation of a robust yet efficient vision-language solver, exemplified by PrismCaptioner-2B with ChatGPT, which delivers impressive results.