LoGra-Med: Long Context Multi-Graph Alignment for Medical Vision-Language Model

Recent research has demonstrated that incorporating long-context significantly enhances LLMs’ ability to process complex inputs and improves instruction-following by retaining more relevant information (Liu et al., 2024c; An et al., 2024; Pawar et al., 2024). Building on this insight, we extend medical instruction-following data by generating long-context paraphrased versions of existing samples in the dataset, providing an additional enriched perspective of the original data.

In particular, a typical instruction sample includes $\{{\bm{X}}_{v},[{\bm{X}}_{q}^{1},\,{\bm{X}}_{a}^{1}],...,[{\bm{X}}_{q}^{L},\,{\bm{X}}_{a}^{L}]\}$ where ${\bm{X}}_{v}$ is an input image, ${\bm{X}}_{q}^{l}$ a question, and ${\bm{X}}_{a}^{l}$ an answer at round $l$ in multi-round $L$ of a conversation. In the medical domain, most of the questions are generic, and the information answer usually covers the question, so we only focus on extending the answer ${\bm{X}}_{a}$. We leverage the GPT API with a prompt to form a longer-context for each ${\bm{X}}_{a}^{l}$ by:

$${\bm{X}}_{ae}^{l}=\mathrm{GPT}\left({\bm{X}}_{q}^{l},{\bm{X}}_{a}^{l},\mathrm{prompt}\right),\ \forall l\in[L].$$

(1)

The details for $\mathrm{prompt}$ are presented in the Appendix. In short, we ask GPT to provide additional explanations for concepts that appeared in the original answer ${\bm{X}}_{a}$ while keeping the content consistent. An example output for ${\bm{X}}_{ae}^{l}$ is illustrated in Figure 3.

For each image ${\bm{X}}_{v}\in\mathbb{R}^{3\times H\times W}$ where $(H,W)$ are the original spatial dimensions. We divide it into a sequence of visual patches ${\bm{U}}=[u_{i}]_{i=1}^{N}$ with $N=(H\times W)/U$ with $U$ as the patch size. Using a pre-trained ViT model $f_{\theta}$, we extract patch-wise features as ${\bm{V}}=f_{\theta}({\bm{U}})\in\mathbb{R}^{N\times d_{v}}$ and apply another projector to map it into the projected embedding ${\bm{Z}}=h_{\phi}({\bm{V}})\in\mathbb{R}^{N\times d}$. We then pool the features from the image patches to define a global description as ${\bm{Z}}_{v}=\mathbb{E}({\bm{Z}})\in\mathbb{R}^{d}$. For each language input ${\bm{X}}_{c}^{l}\in\{{\bm{X}}_{a}^{l},{\bm{X}}_{ae}^{l}\}$ with $c\in\{a,ae\}$, we assume it has $M$ tokens, i.e., ${\bm{X}}_{c}^{l}=[{\bm{x}}_{j}]_{j=1}^{M}\in\mathbb{R}^{M}$, and feed it into the LLM model to extract a set of embedding ${\bm{Z}}_{c}^{l}=g_{\sigma}([{\bm{x}}_{j}]_{j=1}^{M})=[{\bm{e}}_{j}]_{j=1}^{M}\in\mathbb{R}^{M\times d}$. We subsequently concatenate all multi-round $L$ in each single instruction tuning to define ${\bm{Z}}_{c}=\textstyle\frac{1}{L}\sum_{l=1}^{L}\mathbb{E}({\bm{Z}}_{c}^{l})$ which collects average text embedding of original answers ($c=a$) and their longer-context extended versions ($c=ae$) respectively.

Given a batch size of $B$ instruction-tuning samples, we now construct three graphs ${\mathcal{G}}_{v}=({\mathcal{V}}_{v},{\mathcal{E}}_{v})$, ${\mathcal{G}}_{a}=({\mathcal{V}}_{a},{\mathcal{E}}_{a})$, and ${\mathcal{G}}_{ae}=({\mathcal{V}}_{ae},{\mathcal{E}}_{ae})$ representing for visual image features, text embedding encoded by LLM for original answers and their long-context embedding extended by GPT. Specifically, for each triplet pair $\{{\bm{X}}_{v}^{(k)},[{\bm{X}}_{a}^{l}]^{(k)},[{\bm{X}}_{ae}^{l}]^{(k)}\}_{k},\,(k\in[B])$, we add a node representing ${\bm{X}}_{v}^{(k)}$ to ${\mathcal{V}}_{v}$, a node for $[{\bm{X}}_{a}^{l}]^{(k)}$ to ${\mathcal{V}}_{e}$, and finally a node for $[{\bm{X}}_{ae}^{l}]^{(k)}$ to ${\mathcal{V}}_{ae}$. This results in a set of nodes ${\mathcal{V}}_{v}=\{{\bm{X}}_{v}^{(1)},...,{\bm{X}}_{v}^{(B)}\}$; ${\mathcal{V}}_{c}=\{[{\bm{X}}_{c}^{l}]^{(1)},...,[{\bm{X}}_{c}^{l}]^{(B)}\}$ for each $c\in\{a,ae\}$. We equip node-level feature matrices for these graphs using their embedding computed above, i.e., ${\bm{F}}_{v}=\{{\bm{Z}}_{v}^{(1)},...,{\bm{Z}}_{v}^{(B)}\}$, ${\bm{F}}_{c}=\{{\bm{Z}}_{c}^{(1)},...,{\bm{Z}}_{c}^{(B)}\}$. The edges for ${\mathcal{E}}_{v},{\mathcal{E}}_{c}$ afterward can be created through the k-nearest neighbors algorithm given the feature node matrices ${\bm{F}}_{v},\,{\bm{F}}_{c}$. Finally, we can run a message-passing network $m_{\alpha}(.)$ on three built graphs to learn richer node representations. This approach has proven effective for representation learning (Tang et al., 2022; Ju et al., 2024), resulting in aggregated feature-node matrices as $\{\hat{{\bm{Z}}}_{s}^{(1)},...,\hat{{\bm{Z}}}_{s}^{(B)}\}=m_{\alpha}({\bm{F}}_{s},{\mathcal{E}}_{s})$, with $s\in\{v,a,ae\}$.

We first provide background about the second-order graph alignment between two arbitrary graphs ${\mathcal{G}}_{1}=({\mathcal{V}}_{1},{\mathcal{E}}_{1})$ and ${\mathcal{G}}_{2}=({\mathcal{V}}_{2},{\mathcal{E}}_{2})$, which is mentioned as quadratic assignment problem and applied in several problems in vision and graphics to find correspondences between two graph structures under consistent between node features and structure constraints (Zanfir & Sminchisescu, 2018; Haller et al., 2022; Ehm et al., 2024).

We denote by ${\bm{V}}\in\{0,1\}^{|{\mathcal{V}}_{1}||{\mathcal{V}}_{2}|}$, with $|{\mathcal{V}}_{1}|=M$ and $|{\mathcal{V}}_{2}|=N$, the indicator matrix of matched vertices, that is, ${V}_{i,j}=1$ if a vertex $v_{i}\in{\mathcal{V}}_{1}$ is matched with $v_{j}\in{\mathcal{V}}_{2}$ and ${V}_{i,j}=0$ otherwise. That is, ${\bm{V}}$ is a binary matrix with exactly one non-zero entry in each row and column. Similarly, we set ${\bm{\mathsfit{E}}}\in\{0,1\}^{|{\mathcal{E}}_{1}||{\mathcal{E}}_{2}|}$ as the indicator tensor of match edges, that is, ${\mathsfit{E}}_{i,k,j,l}=1$ if ${V}_{i,j}=1$ and ${V}_{k,l}=1$ and ${\mathsfit{E}}_{i,k,j,l}=0$ otherwise. This implies that the tensor ${\bm{\mathsfit{E}}}$ is fully determined by the matrix ${\bm{V}}$, that is, ${\mathsfit{E}}_{i,k,j,l}={V}_{i,j}{V}_{k,l}$. We also define the vertex affinity matrix and edge affinity tensor as ${\bm{A}}^{v}\in\mathbb{R}^{|{\mathcal{V}}_{1}||{\mathcal{V}}_{2}|}$ and ${\bm{\mathsfit{A}}}^{e}\in\mathbb{R}^{|{\mathcal{E}}_{1}||{\mathcal{E}}_{2}|}$, respectively. The set ${\mathcal{A}}({\mathcal{G}}_{1},{\mathcal{G}}_{2})$ indicates for all admissible pairs $({\bm{V}},{\bm{\mathsfit{E}}})$ that encode a valid matching between ${\mathcal{G}}_{1}$ and ${\mathcal{G}}_{2}$.

$\displaystyle{\mathcal{A}}({\mathcal{G}}_{1},{\mathcal{G}}_{2})=\left\{{\bm{V}}\in\{0,1\}^{M\times N}:\sum_{i=1}^{M}{V}_{i,j}=1,\sum_{j=1}^{N}{V}_{i,j}=1\right\}.$

(2)

The second-order graph alignment (SoGA) problem now is defined as:

$\displaystyle\text{SoGA}({\bm{A}}^{v},{\bm{\mathsfit{A}}}^{e})$

$\displaystyle=\operatorname*{arg\,min}_{{\bm{V}}\in{\mathcal{A}}({\mathcal{G}}_{1},{\mathcal{G}}_{2})}\langle{\bm{A}}^{v}+{\bm{\mathsfit{A}}}^{e}\otimes{\bm{V}},{\bm{V}}\rangle=\operatorname*{arg\,min}_{{\bm{V}}\in{\mathcal{A}}({\mathcal{G}}_{1},{\mathcal{G}}_{2})}\sum_{i,j}{A}^{v}_{i,j}{V}_{i,j}+\sum_{i,j,k,l}{\mathsfit{A}}^{e}_{i,j,k,l}{V}_{i,j}{V}_{k,l}.$

(3)

Our aim is to solve the graph alignment between three graphs ${\mathcal{G}}_{v},{\mathcal{G}}_{a}$, and ${\mathcal{G}}_{ae}$ to form a triplet constraint between input image embedding, its original instruction embedding, and the long-context extended version ones. However, solving a structure-aware graph alignment between $K$ domains is computationally expensive. One potential solution is to perform pairwise graph alignments $\binom{K}{2}$ times, as shown in Eq. (3), while applying specific constraints to maintain consistency between correspondences (Bernard et al., 2019; Swoboda et al., 2019). However, this method becomes impractical as $K$ increases, making it unsuitable for larger-scale problems.

Another direction leverages the barycenter concept from optimal transport, which identifies a central distribution that minimizes the weighted sum of Wasserstein distances to the given input distributions. We follow this idea to reformulate the alignment of $K$ graphs into $K$ separate alignments with a barycenter graph. Unlike previous unsupervised methods that estimate the barycenter before aligning, we directly define the barycenter using known triplet pairs across the three graphs. This significantly reduces complexity, making our solver more efficient in LLM settings.

Specifically, we define a new barycenter graph ${\mathcal{G}}_{br}=({\mathcal{V}}_{br},{\mathcal{E}}_{br})$ where ${\mathcal{V}}_{br}=\left\{v_{br}^{(1)},...,v_{br}^{(B)}\right\}$ with $v_{br}^{(k)}=\overline{{\bm{X}}}^{(k)}=\{{\bm{X}}_{v}^{(k)},[{\bm{X}}_{a}^{l}]^{(k)},[{\bm{X}}_{ae}^{l}]^{(k)}\}$ and a correspondence feature node as ${\bm{F}}_{br}=\dfrac{1}{3}\left\{\sum_{s}\hat{{\bm{Z}}}_{s}^{(1)},...,\sum_{s}\hat{{\bm{Z}}}_{s}^{(B)}\right\}$ with $s\in\{v,a,ae\}$. The edge set ${\mathcal{E}}_{br}$ is formed similarly to another graph by running the k-nearest neighbor on feature node ${\bm{F}}_{br}$. We now state the multi-graph alignment as:

$$\text{SoGA}({\bm{A}}_{s}^{v},{\bm{\mathsfit{A}}}_{s}^{e})=\operatorname*{arg\,min}_{{\bm{V}}_{s}\in{\mathcal{A}}({\mathcal{G}}_{s},{\mathcal{G}}_{br})}\sum_{s\in\{v,a,ae\}}\,\langle{\bm{A}}_{s}^{v}+{\bm{\mathsfit{A}}}_{s}^{e}\otimes{\bm{V}}_{s},{\bm{V}}_{s}\rangle,$$

(4)

where ${\bm{V}}_{s}$ is the indicator matrix representing for valid mapping between ${\mathcal{G}}_{s}$ and ${\mathcal{G}}_{br}$, ${\bm{A}}_{s}^{v}\in\mathbb{R}^{|{\mathcal{V}}_{s}||{\mathcal{V}}_{br}|}$ and ${\bm{\mathsfit{A}}}_{s}^{e}\in\mathbb{R}^{|{\mathcal{E}}_{s}||{\mathcal{E}}_{br}|}$ be vertex affinity matrix and edge affinity tensor between ${\mathcal{G}}_{s}$ and ${\mathcal{G}}_{br}$. For e.g., $\left({\bm{A}}_{s}^{v}\right)_{ij}=d\left(\hat{{\bm{Z}}}_{s}^{(i)},\dfrac{1}{3}\sum_{s}\hat{{\bm{Z}}}_{s}^{(j)}\right)$ with $d(.)$ be a distance metric (e.g., cosine distance) measuring similarity between node $i^{th}$ in ${\mathcal{G}}_{s}$ and node $j^{th}$ in ${\mathcal{G}}_{br}$.

To address the NP-Hard nature of aligning each graph to the barycenter graph ${\mathcal{G}}_{c}$, which arises from its combinatorial complexity, we employ efficient heuristic solvers utilizing Lagrange decomposition techniques (Swoboda et al., 2017; Rolínek et al., 2020).

Given $\hat{{\bm{V}}}_{s}=\text{SoGA}({\bm{A}}_{s}^{v},{\bm{\mathsfit{A}}}_{s}^{e})$ be solution obtained from the solver, we aim to learn feature representation for LLMs such that $\hat{{\bm{V}}}_{s}$ be identical to true triplet alignments explicitly indicated by the barycenter graph. By denoting ${\bm{V}}_{s}^{*}$ be an optimal mapping between the graph ${\mathcal{G}}_{c}$ to ${\mathcal{G}}_{br}$, we compute the following total of hamming loss function:

$$\mathcal{L}(\hat{{\bm{V}}}_{s},{\bm{V}}_{s}^{*})=\sum_{s\in\{v,a,ae\}}\langle\hat{{\bm{V}}}_{s},(1-{\bm{V}}_{s}^{*})\rangle+\langle{\bm{V}}_{s}^{*},(1-\hat{{\bm{V}}}_{s})\rangle.$$

(5)

However, computing the gradient of the loss function with respect to the matching problem inputs $\left({\bm{A}}_{s}^{v},{\bm{\mathsfit{A}}}_{s}^{e}\right)$, i.e., $\nabla_{{\bm{A}}_{s}^{v},{\bm{\mathsfit{A}}}_{s}^{e}}\mathcal{L}\left(\hat{{\bm{V}}}_{s},{\bm{V}}_{s}^{*}\right)$, poses a challenge due to the piecewise constant nature of the graph matching objective in Eq. (4) (Pogančić et al., 2020; Rolínek et al., 2020). To address this, we resort to the IMLE techniques (Niepert et al., 2021; Minervini et al., 2023), a method permitting estimate gradients over solutions of the combinatorial optimization problem by taking the difference between solutions of matching problem perpetuated by Gumbel noise.

In particular, given $(\epsilon,\epsilon^{\prime})\sim\mathrm{Gumble}(0,1)$ and for each $s\in\{v,a,ae\}$, we compute:

	$\displaystyle\left(\dfrac{\partial\mathcal{L}}{\partial{\bm{A}}_{s}^{v}},\dfrac{\partial\mathcal{L}}{\partial{\bm{\mathsfit{A}}}_{s}^{e}}\right)$	$\displaystyle\approx\tilde{{\bm{V}}}_{s}-\text{SoGA}\left({\bm{A}}_{s,\lambda}^{v},{\bm{\mathsfit{A}}}_{s,\lambda}^{e}\right)\text{ where }\tilde{{\bm{V}}}_{s}=\text{SoGA}\left({\bm{A}}_{s}^{v}+\epsilon,{\bm{\mathsfit{A}}}_{s}^{e}+\epsilon^{\prime}\right),$		(6)
	$\displaystyle\left({\bm{A}}_{s,\lambda}^{v},{\bm{\mathsfit{A}}}_{s,\lambda}^{e}\right)$	$\displaystyle=\left({\bm{A}}_{s}^{v}+\epsilon,{\bm{\mathsfit{A}}}_{s}^{e}+\epsilon^{\prime}\right)-\lambda\nabla_{\tilde{{\bm{V}}}_{s}}\mathcal{L}(\tilde{{\bm{V}}}_{s},{\bm{V}}_{s}^{*}),\text{ with }\lambda\text{ is a step size.}$		(7)

In this section, we present theoretical insights into the graph-matching problem outlined in Eq. (3). Specifically, we demonstrate that once the optimal matching between two graphs is established, it defines a valid metric distance. Additionally, the geodesic path (i.e., the shortest path distance) connecting the two graphs in the manifold space can be derived based on the computed matching alignments. We define a discrete between two graphs given a solution of matching alignment as:

$\displaystyle d_{\text{SoGA}}({\mathcal{G}}_{1},{\mathcal{G}}_{2})$

$\displaystyle=\min_{{\bm{V}}\in{\mathcal{A}}({\mathcal{G}}_{1},{\mathcal{G}}_{2})}\Big{(}\sum_{i,j}{A}^{v}_{i,j}{V}_{i,j}+\sum_{i,j,k,l}{\mathsfit{A}}^{e}_{i,j,k,l}{V}_{i,j}{V}_{k,l}\Big{)}.$

(8)

We consider labelled graphs as tuples of the form ${\mathcal{G}}=({\mathcal{V}},{\mathcal{E}},{\mathcal{L}}_{f},{\mathcal{L}}_{s})$, where the labelling function ${\mathcal{L}}_{f}:{\mathcal{V}}\mapsto\mathcal{F}$ assigns each vertex $v_{i}\in{\mathcal{V}}$ to a feature $f_{i}={\mathcal{L}}_{f}(v_{i})$ in some feature space $(\mathcal{F},d_{f})$. Similarly, we denote ${\mathcal{L}}_{s}:{\mathcal{V}}\mapsto\mathcal{S}$ as a structure function which links each vertex $v_{i}\in{\mathcal{V}}$ with its structure information $s_{i}={\mathcal{L}}_{s}(v_{i})$, e.g.,  edge information, in some structure space $(\mathcal{S},d_{s})$. By associating a weight to each vertex, we allow the graph ${\mathcal{G}}$ to be represented by a fully supported mixing measure $\mu=\sum_{i=1}^{N}w_{i}\delta_{(f_{i},s_{i})}$ over the product between feature space and structure space ${\mathcal{F}}\times{\mathcal{S}}$. Notably, $\mu$ is not necessarily a probability measure as the summation of its weights can be different from one. We have the vertex affinity matrix between two graphs as ${\bm{A}}^{v}\in\mathbb{R}^{M\times N}$, where ${A}^{v}_{i,j}=(d_{f}(f_{i},f_{j})){i,j}$. Structural similarity is measured by pairwise distances within each graph, represented by ${\bm{\mathsfit{A}}}^{e}\in\mathbb{R}^{|{\mathcal{E}}_{1}||{\mathcal{E}}_{2}|}$, with ${\mathsfit{A}}^{e}_{i,j,k,l}=|d_{s}(s_{i},s_{k})-d_{s}(s_{j},s_{l})|$, where $d_{s}(\cdot)$ models node distance, such as the shortest path. We then define the space of all structured graphs $({\mathcal{F}}\times{\mathcal{S}},d_{f},\mu)$ over a metric feature space $({\mathcal{F}},d_{f})$ as ${\mathbb{S}}({\mathcal{F}})$, where $(\mathcal{S},d_{s})$ is a metric structure space and $\mu=\sum_{i=1}^{N}w_{i}\delta_{(f_{i},s_{i})}$ is a mixing measure over ${\mathcal{F}}\times{\mathcal{S}}$.

Theorem 1 is proved in Appendix A.1. Intuitively, we can compare two graphs, ${\mathcal{G}}_{1}$ and ${\mathcal{G}}_{2}$, described by their mixing measures $\mu_{1}=\sum_{i=1}^{M}w_{1i}\delta_{(f_{1i},s_{1i})}$ and $\mu_{2}=\sum_{j=1}^{N}w_{2j}\delta_{(f_{2j},s_{2j})}$, with uniform vertex weights (i.e.,  $w_{1i}=1,\forall i\in[M]$, $w_{2j}=1,\forall j\in[N]$) and shortest path structure matrices. The SoGA distance is zero if and only if there exists a one-to-one mapping between the graph vertices that preserves both shortest paths and features and both graphs have the same number of vertices.

Model architectures. We use the LLaMA large language model (Touvron et al., 2023), the CLIP-ViT-L-Patch14 visual encoder (Radford et al., 2021), and an MLP projection similar to LLaVA 1.5 (Liu et al., 2024a). Stage 1 follows the standard LLaVA-Med (Li et al., 2024) setup, while stage 2 incorporates our multi-graph alignment with autoregressive training. For multi-graph alignment, a 2-layer graph convolutional network is applied to the output of the Projection and LLM Decoder (handling both image and text modalities). We train for 1 epoch in stage 1 and 3 epochs in stage 2 using the same dataset as LLava-Med. The model is optimized using Adam (Kingma, 2014) with CosineAnnealingLR scheduler and learning rates of $2e-3$ and $2e-5$ for stages 1 and 2, respectively.

Pre-training data. We follow the same dataset being used in LLaVA-Med (Li et al., 2024). For stage 1, there are 600K image-text pairs filtered from PMC-15M. They are all converted to instruction-following data with simple instructions for describing the image. For stage 2, there are 60K image-text pairs extracted from PMC-15M with 5 modalities: CXR (chest X-ray), CT (computed tomography), MRI (magnetic resonance imaging), histopathology, and gross (i.e., macroscopic) pathology. Then, the authors from LLaVA-Med (Li et al., 2024) use language-only GPT-4 to generate multi-round questions and answers in a tone as if it could see the image to convert these pairs to instruction-following format.

Running-time. We train LoGra-Med using 4 A100-GPUs per with 80GB for both stages and complete the training process for stage 1 in 7 hours and for stage 2 in 7.5 hours. With original LLaVA-Med (version 1.5) (Li et al., 2024), the training process for stage 1 finishes in 6.5 hours, and for stage 2 finishes in 7 hours. In total, we need extra one more hour to complete the whole pre-training process compared to the LLaVa-Med.

We begin by presenting the data-intensive nature of autoregressive training. To assess this, we used LLaVA-Med, a state-of-the-art multimodal large language model in the biomedical domain. LLaVA-Med follows a two-stage in their training process: Stage 1 aligns image-text tokens with biomedical concepts, and Stage 2 fine-tunes the model for instruction-based tasks. We pre-trained LLaVA-Med on varying data amounts (10%, 40%, 70%) and used the checkpoints to fine-tune the visual question answering downstream task. As shown in Figure 1, performance on the VQA-RAD dataset sharply declines when using 10% pre-training data compared with fully-training on 100%. To the best of our knowledge, this is the first time that the auto-regressive mechanism’s data dependency issue in medical-MLLM has been revealed. This evidence indicates that without using enough instruction-tuning data, the auto-regressive creates a weak connection between visual features and text embeddings in LLM.

To tackle this issue, our LoGra-Med enhances image-text alignment through the semantic constraints between input images, their instruction response, and the long-context extended contexts. We experiment with LoGra-Med under the same settings as LLaVA-Med, pretraining on varying data sizes and utilizing the checkpoints for instruction fine-tuning. Figure 1 demonstrates that LoGra-Med effectively learns the biomedical concept alignment task, producing strong pretraining checkpoints for downstream fine-tuning while using only 10% of the data. For instance, while LLaVa-Med is at $52.39$ with $10\%$, LoGra-Med reaches $72.52\%$, which is approximately LLaVa-Med pre-trained on $100\%$. Across 10%, 40%, and 70% data usage, LoGra-Med shows stable performance with consistent improvement, positioning it as a competitive model compared to other medical-MLLM like BioMed-GPT (Zhang et al., 2023a) or Med-MoE (Jiang et al., 2024). This suggests that the multi-graph alignment strategy effectively mitigates the data-intensive demands of autoregressive mechanisms in mLLMs.

Datasets. We evaluate pre-trained models on three prominent biomedical VQA datasets: VQA-RAD (Lau et al., 2018), SLAKE (Liu et al., 2021), and PathVQA (He et al., 2020). VQA-RAD includes 3,515 questions across 315 radiology images, while SLAKE contains 642 radiology images from various body parts and over 7k QA pairs. PathVQA, focused on pathology, features 5k images and 32.8k questions. All datasets include open-ended (e.g., what, why, where) and closed-ended (yes/no or two-option) question types. We provide more details in the Appendix.

Baselines. We compare four approaches, including InfoNCE-based methods (Khan & Fu, 2023; Liu et al., 2023), SigLIP (Zhai et al., 2023), PLOT (Chen et al., 2022), and VLAP (Park et al., 2024). Among this, SigLIP adapts the Sigmoid loss on image-text pairs to break the global view of the pairwise similarities for normalization, resulting in scaling in large batch size. PLOT defines optimal transport as a distance between visual image patches and text embedding. In contrast, VLAP uses assignment prediction to bridge the modality gap between the visual and LLM embeddings. We train the baselines under the same settings as LoGra-Med with varying pre-training data rates and compare their performance on downstream tasks.

We now compare LoGra-Med pre-trained with $100\%$ data against other medical foundation models, each trained on varying datasets and employing different architectures or model sizes.

Baselines. We compare with eight competitors, both generic or medical foundation models, including LLaVA (Liu et al., 2024b), LLaVA-Med (Li et al., 2024), Med-Flamingo (Moor et al., 2023), Med-Dr (He et al., 2024), Biomed-GPT (Zhang et al., 2023a), M2I2 (Li et al., 2023b), GPT-4o (Achiam et al., 2023) and Med-MoE (Jiang et al., 2024). Whilst LLaVA and GPT-4o have no medical background, the others are pre-trained on a variety of biomedical knowledge. With the exception of LLaVa, which we reproduced, the results for the other baselines are taken from the literature. Moreover, we also present an enhanced version, LoGra-Med + DCI, which integrates multi-scale visual features from vision encoders (Yao et al., 2024), potentially benefiting medical image analysis by considering both local (detailed) and global (contextual) features.