Bridging Text and Molecule: A Survey on Multimodal Frameworks for Molecule

The Simplified Molecular Input Line Entry System (SMILES) is the most frequently used sequential descriptor of small molecules, which maps atoms, bonds, and special structures with ASCII symbols. The molecular sequences can be tokenized like text sequence, and Transformer Vaswani et al. (2017) can be employed for molecular encoding Zeng et al. (2022); Liu and others (2023b).

The topology structure of molecules can be naturally modeled by graph, with atoms as nodes and bonds as edges. GNNs Kipf and Welling (2017) can be used to learn local and global representations of molecular graphs. Recently, the molecular representations that are pre-trained with GNNs structure have shown competitive results in various downstream tasks Liu et al. (2022), demonstrating the effectiveness of graph descriptors.

2D molecular graphs have limitations in capturing the spatial information within molecules. For example, chiral molecules cannot be distinguished through most of the 2D graph. The geometry information of conformers (e.g., torsional angles, bond length) holds direct relation with molecular properties. In 3D geometry, atoms are associated with their coordinates with features expressed in high-order tensors to ensure geometric symmetries and expressiveness. Many studies concentrate on designing equivariant GNNs to accurately model the interaction between atoms Batzner and others (2022).

A protein can be viewed as a combination of 20 types of amino acids, making it possible to be expressed as amino acid sequences in a similar manner to molecules. The amino acid sequence captures the co-evolutionary information and plays a vital role in protein folding and function. Transformer-based models, commonly referred as protein language models (PLMs), use similar architecture in Natural Language Models to featurize protein for prediction or editing tasks.

Protein functions are primarily determined by their folded structures Jumper et al. (2021). To better capture structural information, proteins can be represented as a residue-level relation graph, where nodes represent the alpha carbon of residues and edges represent connectivity between residues and amino-acids. We can also employ GNNs such as Massage Passing Neural Networks (MPNNs) for protein graph encoding Dauparas et al. (2022).

A single-stream architecture assumes that the latent space of molecules and text share similar semantic meaning. Under this circumstance, molecules are treated as a specialized language and expressed by sequential descriptors such as SMILES. After tokenization, the molecular and textual tokens are typically fed into an encoder-decoder language model, such as T5 Raffel et al. (2020), for multi-language pre-training. KV-PLM Zeng et al. (2022) and MolXPT Liu and others (2023b) use byte-pair encoding (BPE) to tokenize SMILES and replace all molecular names in sequence with SMILES tokens, making these “wrapped” sequences as training data. BioT5 Pei et al. (2023) use separate vocabularies for molecules, proteins, and texts to avoid misunderstanding of tokens that may have the same expression but originate from different semantic spaces. Notably, GIMLET Zhao et al. (2023a) serializes molecular graphs as node sequences and introduce position embedding to jointly encode nodes with related text tokens. This setting not only maintains the inductive bias at the graph level but also avoids introducing an extra graph encoding block and keeps molecular features independent of text.

Models with multi-stream architecture employ intra-modality processing for text and molecules. GNNs are generally adopted as molecular encoders to encode structural information. The representations are then projected into textual latent space by a linear layer, or projected with cross-attention mechanism for cross-modal fusion and alignment. For instance, Abdine et al. (2023) fuse protein sequence and protein graph features by element-wise addition and use them as input of cross-attention module to adapt with text. Xu et al. (2023) choose both text and protein representation as keys and apply two separate cross-attention modules to produce fused-text and fused-protein representations.

Q-Former Li and others (2023) is a representative architecture in vision-language modeling that leverages cross-attention layers to bridge the modality gap. Similarly, Li et al. (2024); Liu et al. (2023d); Zhang et al. (2023) adopt Q-Former to connect molecular graph with text and extract text-related molecular features with a learnable query. Particularly, Liu et al. (2023a) propose GIT-Former which can be viewed as a variation of Q-Former with additional input modalities from molecular images and sequences.

The contrastive learning (CL) task between molecule and text aims to align multi-modal representations by enhancing the correlation between matched molecule-text pairs. The contrastive learning objective pushes the embeddings from matched text and molecules closer in latent space while enlarging the distance between pairs from different molecules. The CL task will enhances the model with cross-modal retrieval and matching ability. Here, we present the expression of commonly used InfoNCE Oord et al. (2018) loss:

where $\tau$ is the temperature coefficient. In order to facilitate the convergence, a trainable linear projector can be used to minimize the modality gap before the contrastive learning Liu and others (2023a).
Although contrastive learning is an effective approach for cross-modal molecule-text alignment, it has some domain-specific limitations. One issue is that the structural information of molecules will lost in the process of encoding. Additionally, limited number of molecule-text pairs also has impacts on the alignment result. Motivated by the molecular graph augmentation methods You et al. (2020), we could adopt augmented graphs to construct molecule-text pairs, as the augmented graphs remain similar semantics information to the original graphs, such as properties and structures. Applying a contrastive loss between the embeddings of the original graph and the augmented graph also ensures this consistency. MoMu Su et al. (2022) introduces two augmented graphs with node dropping and sub-graph extraction to construct matched pairs with text descriptions. MolLM Luo et al. (2024) follows the same augmentation rules in MoMu and introduces two extra augmentations which are chemical transformation and motif removal. These augmentations increase the size of training data, making the alignment process more robust.

Molecule-text matching (MTM) aims to predict whether a molecule-text pair is matched or not. It is defined as a binary classification task with the following loss function:

where $(m_{i},t_{i})$ denote matched molecule and text pair. The MTM task enables the model with the retrieval ability and refines the alignment between text and molecule. Liu et al. (2023d); Li et al. (2024) use MTM in the first-stage training of Q-Former. Liu et al. (2023a) extend the MTM to cross-modal matching which performs matching tasks between text, graphs, and images.

Conditional generation (CG) aims to generate tokens based on given conditions or constraints. Tasks such as molecule captioning and text-based molecule generation all fall into this category. Conditional generation enables models to learn complex mapping rules between text and molecules. It is adaptable for T5 architecture where all molecular tasks are transformed into text-to-text generation format. The objective function can be written as:

where $u_{i}$ is the $i$-th token and $C$ denotes the generation condition which may be referred to as a molecule graph or text description depending on the task.

As discussed in section 3, modeling languages and molecules may share similarities. Under this assumption, masked language modeling as a prevalent pre-training task for LLMs can also be used for training molecule sequences or wrapped sequences. During the pre-training stage, the models are trained to predict the masked components using the remaining context. The training objective is defined by cross-entropy

where $\mathcal{M},T\backslash\mathcal{M},T$ represent the masked tokens, unmasked tokens, tokenized text and molecules separately. This self-supervised pre-training task can enhance the contextual comprehension of the model, improving performance in many downstream tasks. For MLM, there are two types of masking: token masking represented by BERT Devlin and others (2018) and its variants, and span masking represented by T5 Raffel et al. (2020). According to Raffel et al. (2020), span masking is more efficient. Zeng et al. (2022) randomly mask tokens from both molecules and text in wrapped text sequences. Edwards et al. (2022); Pei et al. (2023); Rubungo et al. (2023); Qian et al. (2023) all adopt span masking for MLM with corresponding T5 backbone to enhance the downstream translation tasks between molecule and text.
Similarly, the protein language models (PLM) introduce masked protein modeling (MPM) by masking residues in protein sequences. ProtST Xu et al. (2023) not only uses MPM for protein encoder pre-training, it also utilizes fused text and protein representations to predict the type of masked residues and language tokens. Before entering the PLM and biomedical language model, 15% of residues in protein sequence and 15% of word tokens in text are masked and then embedded. Two MLPs are trained with MPM objective to recover the masked components in text and residues.

Different from the autoencoder (AE) language models such as BERT and T5 which adopt MLM as the training objective, the autoregressive models represented by GPT Yenduri et al. (2023) are trained with Casual Language Modeling (CLM). The objective of CLM is to predict the next token in a sequence in a left-to-right direction. The objective function can be written as

where $n_{i}$ and $k$ represent the number of tokens and context length. Transformation of molecular tasks into text generation helps the CLM seamlessly integrate into the training and instruction-tuning process. Liang et al. (2023); Cao et al. (2023); Zhang et al. (2023). We will discuss the detail of instruction-tuning and adaptation of tasks in the following sections.

De novo generation in molecule design includes molecule captioning which generates a description of given molecules and text-guided de novo generation which generates molecules from scratch by the textual guidance. Models with single-stream architecture have the privilege of performing translation between text and molecule, owing to the encoder-decoder structure and text-to-text task format. Raffel et al. (2020). Apart from the translation-based methods. Liu et al. (2023c) propose a protein design framework with a multi-stream encoder. In text-guided protein generation task, the description is first encoded by the aligned text encoder. Then a facilitator module which is parameterized by a multi-layer perception is used to learn the transformation from encoded text to protein representation. The resulting protein representation is then fed into a trained generative decoder to generate protein sequences.

Molecule editing seeks to optimize current molecules with desired properties. Within the drug discovery pipeline, text-guided editing finds application in lead optimization tasks and proves valuable for decomposing multi-objective lead optimization Liu et al. (2023c). Drawing inspiration from the success of few-shot text-to-image generation, text description can simplify the complexity of the target chemical space in the generation process. Simultaneously, personalized generation enhances drug editing by introducing high flexibility. As mentioned above, the latent space alignment establishes a unified latent space where features possess semantic meaning in both structure and text. Building upon this approach, Liu and others (2023a); Liu et al. (2023c); Tang et al. (2023) use latent optimization methods to sample a latent representation close to both text and molecule in latent space. Then this latent code is fed into a decoder which is usually a trained molecule generation model to produce optimized molecules. Kim et al. (2023a) proposes hierarchical textual inversion which introduces intermediate and detail tokens to represent SMILES, aiming to capture cluster-level and molecule-level features. The interpolation sampling can benefit from this hierarchical design with high generation diversity.

Reaction prediction is a challenging but fundamental task in chemistry and biology. The chemical reaction process can be seen as a mapping between a set of reactants to a set of products with specific reaction conditions. Based on this assumption, there are three main tasks in reaction prediction, which are product prediction, reaction condition prediction, and most importantly, retrosynthesis prediction. Text can supply information in complex reaction mechanisms and reaction templates which GNN-based methods often fail to capture. Qian et al. (2023) retrieve reaction-related text and concatenate with input similes to enhance the retrosynthesis prediction. As described in 4.3, we can also involve LLMs in reaction prediction via prompting engineering. For example, Shi et al. (2023) use GPT-4 to predict reaction products with the aid of in-context reaction examples and reaction prediction model.

According to Bran et al. (2023), the automation level in chemistry is relatively low compared to other domains. Despite LLMs may have difficulties in comprehending chemistry principles, they have demonstrated significant capability in understanding human instructions and organizing information based on extensive training corpora AI4Science and Quantum (2023). Consequently, LLMs have the potential to become intelligent assistants to automatically arrange research with the help of professional tools and software. Liu et al. (2023b) design a drug editing agent with conversational interaction. The agent can receive human feedback to retrieve candidate drug molecules from the database with desired properties. Boiko et al. (2023) develop a “Coscientist” based on GPT-4 similar to ChemCrow Bran et al. (2023) which can autonomously design and execute chemical research.