From Efficient Multimodal Models to World Models: A Survey

World models are currently one of the hottest research directions in the AI field. From OpenAI to Meta, major AI companies are striving to develop world models. The concept of world models can be traced back to the fields of reinforcement learning and robotic control. Traditionally, reinforcement learning algorithms rely on agents learning through trial and error in real environments, which is not only costly but sometimes infeasible. To overcome these limitations, researchers began exploring methods for simulation and learning within internal environments. Jurgen et al.[1] have described a method for quickly training through unsupervised environments using generative recurrent neural networks (RNN) to compress spatiotemporal representations and simulate common reinforcement learning environments. Jurgen et al. referred to this as a world model. In AI research, the proposal of world models aims to distinguish this direction from another research focus: agents.

World models gained widespread attention thanks to Yann LeCun’s work[2]. Yann LeCun mentioned that human or animal brains seem to run a simulation of the world, which he called a world model. This model helps humans and animals make predictions about their surroundings. LeCun provided an example: a baby learns basic knowledge by observing the world in the first few months after birth, such as understanding gravity when seeing an object fall. This ability to predict what will happen next comes from common sense, which LeCun believes is the essence of intelligence. The Sora model is a model developed by OpenAI for generating video. It utilizes multimodal learning techniques to generate realistic video content by combining text and image data. In recent studies, OpenAI defined Sora in their report as a world simulator capable of generating videos and considered Sora’s technology a promising approach to building a general world model. We will introduce the differences between the two main routes currently being explored: the multimodal large models developed by Meta under Yann LeCun’s guidance and OpenAI’s GPT series.

In summary, we can clearly define a world model as shown in Figure 1. A world model refers to a model that can predict and simulate environmental state changes by learning from various data in the environment. Unlike conventional data testing scenarios where data does not change, a world model’s data can change independently, even generating data not in the test dataset. The core function of a world model lies in counterfactual reasoning, which allows it to infer the outcomes of decisions not encountered before. AI researchers’ pursuit of world models aims to achieve this counterfactual reasoning, a natural ability of humans that current AI lacks.

Currently, there are two main routes to developing world models: autoregressive methods and JEPA (Joint Embedding Predictive Architecture) methods. Autoregressive models hold a significant place in the generative field, with notable representatives including the GPT series and Sora. These models, based on Transformer architecture[3], generate data step-by-step, with each output depending on the previous hidden state. This incremental generation allows the model to capture contextual information, producing coherent and logical sequences. The autoregressive model possesses a robust contextual understanding capability and is facile to train, thus establishing itself as the predominant approach in the field of world modeling. By capitalizing on previously generated content during generation, autoregressive models demonstrate adeptness in comprehending and maintaining contextual consistency, thereby yielding more coherent and meaningful output. The training process for autoregressive models is relatively straightforward, involving step-by-step prediction and optimization based on known sequential data, which contributes to their commendable performance when trained on large-scale datasets. While autoregressive models excel in natural language processing tasks, generating high-quality text paragraphs through pre-training and fine-tuning, critics argue that these models lack real-world common sense, obscured by vast amounts of information. For instance, a baby learns how the world works and can predict outcomes with little practice compared to the extensive training data required for large language models.

In response, Meta proposed the JEPA framework. The core idea of JEPA is hierarchical planning, a method of decision-making and control that is especially suited to handling complex tasks and large-scale problems. This approach involves breaking down problems into multiple levels, each addressing sub-tasks at different levels of abstraction, simplifying the overall problem-solving process. LeCun illustrated this with an example: to travel from New York to Beijing, one must first get to the airport, then take a flight to Beijing, with the overall cost function representing the distance from New York to Beijing. Solving this involves decomposing the task into millisecond-level control, finding the action sequence that minimizes the predicted cost. He believes all complex tasks can be accomplished through such hierarchical methods, with hierarchical planning being the most significant challenge.

JEPA models extract abstract representations of the world state through a series of encoders and use different levels of world model predictors to forecast various states at different time scales. Inspired by the human brain’s ability to understand and react to the environment in a hierarchical manner, JEPA uses a layered architecture to break down complex tasks into multiple levels, each handling sub-tasks at different abstraction levels. This approach enables JEPA to efficiently capture and predict changes in complex dynamic systems, improving the model’s handling of long-time-span and multi-scale data. Its unique hierarchical prediction mechanism not only enhances understanding and prediction accuracy of environmental states but also increases adaptability and robustness in dealing with large-scale, diverse data, showcasing significant advantages in many practical applications.

In summary, we can summarize the route to the world model into two, rules and data drivers.

Regardless of the route to world models, multimodal models are an indispensable part. Multimodal models refer to machine learning models capable of processing and understanding data from different modalities, such as images, text, audio, and video [4, 5]. Human interaction with the real world involves multiple modalities of information, including language, vision, and audio. Therefore, world models must handle and understand multiple forms of data, meaning they must have multimodal understanding capabilities. Additionally, world models simulate dynamic environmental changes to make predictions and decisions, requiring robust multimodal generation capabilities[6]. To put it simply, the world is multimodal, and the world simulator must be able to accept and generate multimodal information. In essence, world models are general-purpose models (General-Purpose Models).

The research on multimodal models can be broadly categorized into several technical approaches: alignment, fusion, self-supervision, and noise addition. Alignment-based methods map data from different modalities to a common feature space for unified processing. Fusion methods integrate multimodal data at different model layers to fully utilize information from each modality. Self-supervised techniques pre-train models on unlabeled data, enhancing performance across various tasks. Noise addition enhances model robustness and generalization by introducing noise into the data.

Combining these techniques allows multimodal models to demonstrate strong capabilities in handling complex real-world data. They can understand and generate multimodal data, simulate and predict environmental changes, and aid agents in making more precise and effective decisions. Thus, multimodal models play a crucial role in developing world models, marking a key step towards general artificial intelligence (General AI). The following sections will detail the technical routes of multimodal models.

In Section 2, we introduce the fundamental technologies of basic architectures. In Section 3, we will introduce the optimization technology of model architecture. Section 4 covers specific technologies of multimodal models. Section 5 compares and contrasts different routes of multimodal models. Finally, Section 6 outlines the potential development paths for multimodal models towards becoming world models.

The Transformer architecture is currently one of the most popular deep learning model architectures, especially for natural language processing (NLP) and computer vision (CV) tasks. The Transformer is a deep learning model designed for handling sequential data, employing an attention mechanism to model long-range dependencies within sequences [4]. Unlike traditional Recurrent Neural Networks (RNNs), the Transformer processes sequences without relying on their order, utilizing self-attention to simultaneously focus on all positions in the sequence, thereby greatly enhancing parallel computation efficiency. The Transformer consists of an encoder and a decoder, where the encoder maps the input sequence into a continuous representation space, and the decoder generates the output sequence based on this representation. Each layer of the encoder and decoder includes a multi-head self-attention mechanism and a feed-forward neural network, stabilized by residual connections and layer normalization. Due to its efficient parallel computation and powerful representation learning capabilities, the Transformer has achieved remarkable success in natural language processing and other tasks requiring sequence data processing. However, the self-attention mechanism of Transformers has high computational complexity when processing long sequences, limiting its efficiency in some applications. To address this issue, researchers have proposed various methods to challenge the Transformer architecture. Linear Attention simplifies the computation of self-attention, reducing its time and space complexity from $O(N^{2})$ to $O(N)$. Key models include Performer[7], Linformer[8], and Linear Transformers[9]. These models can efficiently handle long-sequence data, reducing computational resource consumption. Additionally, Grouped-query Attention and Multi-query Attention are important attention mechanism variants. The former balances multi-head and multi-query attention by sharing a set of keys and values among groups of query heads, while the latter simplifies computation by sharing the same key and value for all query heads, thus improving efficiency. Grouped Query Attention and Multi-query Attention excel in reducing the size of key-value pairs during inference, significantly improving throughput. Multi-Query Attention shares key-value pairs among multiple heads, achieving a 30-40% reduction in throughput. Grouped Query Attention groups queries, sharing key-value pairs within groups, achieving results comparable to Multi-Query Attention in both efficiency and performance. MQA and GQA are used in the well-known open-source large language models Llama-2 and Llama-3[10, 11, 12].

At the block level, optimization methods include Compact Architecture, which reduces layers and parameters for a compact model structure, lowering computational costs; Pruning[13], which reduces redundant parameters through pruning techniques, enhancing computational efficiency; Knowledge Distillation[14], which extracts knowledge from a large teacher model and applies it to a smaller student model, significantly reducing model complexity and computational resource requirements; and Quantization[15], which converts model parameters from high-precision floating points to lower-precision formats, further reducing computational and storage costs. These optimization methods collectively aim to enhance the efficiency and performance of Transformers, enabling them to process and integrate data from different modalities more efficiently in multimodal tasks.

Additionally, there are architectural approaches challenging the Transformer model, such as Gated Convolution[16] or Gated MLP[17], Recurrent Models[18], State Space Models (SSMs)[19, 20], H3[21], RWKV[22], Mega[23], Yan, JEPA[24], and the notable Mamba[25] and Mamba 2[26]. Gated Convolution introduces gating mechanisms in convolutional neural networks (CNNs), enhancing the model’s ability to capture local and long-range dependencies while reducing computational load. Recurrent models like LSTM and GRU capture temporal dependencies in sequences through their recursive structures[27, 28], overcoming the vanishing gradient problem in traditional RNNs. State Space Models explicitly model the relationship between system states and observations, providing a flexible framework for handling time-series data, including State Space Models, H3, Mamba and Mamba 2.

These approaches not only offer new theoretical insights but also demonstrate their advantages in practice. This chapter will provide detailed introductions to these basic techniques in general architectures, focusing on the most representative and mainstream approaches.

Multi-head attention mechanisms are a core component of Transformers, capturing dependencies between different positions in the input sequence through parallel computation of multiple attention heads. However, the standard multi-head attention mechanism has high computational complexity, prompting researchers to propose various variants to optimize its performance.

The standard Transformer model faces efficiency bottlenecks when processing long sequences, with the time and space complexity of its self-attention mechanism being quadratic in sequence length $O(n^{2})$. This issue arises from the Softmax operation in the attention mechanism. As shown in Figure 2, without Softmax, attention computation simplifies to three matrix multiplications, which are associative, allowing the calculation of $K^{T}V$ first, followed by left-multiplying by $Q$. This reduces complexity from $O(n^{2})$ to linear $O(n)$, the core idea of Linear Attention.

Removing Softmax, the attention computation’s complexity can drop to linear $O(n)$. Scaled-Dot Attention essentially weights $V$ with $QK^{T}$. Therefore, a generalized definition of attention can be proposed:

Here, $f$ is a general function, approximating the Softmax operation. To fit Softmax, $f$ must ensure non-negativity $f\geq 0$. This generalized attention form is known as Non-Local Networks in computer vision.

If elements of $Q$ and $K$ are non-negative, their dot product is naturally non-negative. This suggests introducing kernel functions. By adding a non-negative activation function $\phi$ to $Q$ and $K$:

Where $\phi$ is a non-negative activation function, such as $\text{ReLU}(x)$. This method, termed the kernel method, is discussed by A Katharopoulos et al[9]. Performers estimate conventional (softmax) full-rank attention Transformers using linear space and time complexity while retaining provable accuracy, without relying on sparsity or low-rank priors.

Another approach utilizes Softmax’s properties. In Efficient Attention[29], $Q$ normalized along dimension $d$ and $K$ along dimension $n$ naturally satisfy normalization conditions. Thus, Softmax is applied separately to $Q$ and $K$:

This form is a special case of the generalized attention definition. Additionally, sparse attention methods[30], such as Sparse Attention by OpenAI[31], reduce computation by retaining values in local regions only, forcing most attention values to zero. After special design, the non-zero elements of the attention matrix are $O(n)$, achieving linear-level attention.

Reformer[32], another notable improvement, reduces attention complexity to $O(n\log n)$ by using locality-sensitive hashing (LSH) to find the largest attention values and compute only those, achieving sparse attention. Moreover, Reformer redesigns the backward propagation process by constructing reversible feed-forward networks (FFN), reducing memory usage. Despite solving sparse attention’s first drawback, Reformer remains complex, especially LSH-based attention and reversible network backward propagation.

Performers[7] adopt a novel fast attention method, Fast Attention Via Positive Orthogonal Random features (FAVOR+):

Equivalent to the above attention, the scaling factor $\sqrt{d_{k}}$ simplifies $A$. Fast attention (FA) maps $Q$ and $K$ through a $\phi$ function to $Q^{\prime}$ and $K^{\prime}$, approximating $A$ as their product:

$\phi$ maps matrix row vectors. FAVOR+ simulates beyond softmax other kernelizable attention mechanisms effectively. This capability is crucial for accurately comparing softmax with other kernels on large-scale tasks, aiding in finding optimal attention kernels. Performers are fully compatible with conventional Transformers, offering strong theoretical guarantees: unbiased or nearly unbiased attention matrix estimation, unified convergence, and low estimation variance.

Linformer[8] improves self-attention with low-rank matrix approximation, reducing complexity to linear $O(n)$. Linformer retains the original Scaled-Dot Attention form but projects $Q$ and $K$ to low-dimensional space using $n\times k$ matrices before attention, reducing computation. While Linformer excels in some tasks, its performance on long-sequence tasks remains to be verified. Moreover, Linformer faces challenges in autoregressive generation tasks due to the projection process combining entire sequence information, complicating causal masking.

Multi-Query Attention and Group-Query Attention are notable variants. These methods optimize the attention computation process, reducing computational complexity and memory consumption while maintaining or enhancing model performance. Multi-Query Attention (MQA) shares keys and values among all attention heads, computing independent queries for each head, thus lowering complexity and memory usage. In MQA, all attention heads share the same key and value, differing only in queries:

Here, $Q_{i}$ is the query for the $i$th attention head, $K$ and $V$ are shared keys and values. This significantly reduces the number of matrices to compute and store, lowering computational and memory complexity.

As shown in Figure 3, Group-Query Attention differs from MQA by grouping attention heads, with each group sharing the same keys and values, while heads within each group have independent queries. This method reduces complexity while maintaining flexibility:

Here, $Q_{i,j}$ is the query for the $j$th attention head in the $i$th group, $K_{i}$ and $V_{i}$ are the shared keys and values for the $i$th group. This group-sharing approach provides more flexibility and expressiveness while reducing computational and memory complexity.

Additionally, down-sampling techniques such as pooling or using strided 1D convolutions reduce sequence length. IBM’s PoWER-BERT[33] and Google’s Funnel-Transformer[34] improve model efficiency through gradual down-sampling. Although these techniques significantly reduce complexity, they can impact the model’s generative capability.

Overall, modifying attention forms or structures effectively reduces computational complexity while retaining or enhancing performance. These variants offer more efficient solutions for handling long-sequence data and provide new ideas and methods for multimodal task research.

Model compression aims to reduce the number of parameters and computational load in deep neural networks, enhancing efficiency and reducing storage requirements. Pre-trained deep neural network models often face over-parameterization, where only about 5% of the parameters are effective. Model compression techniques include frontend and backend compression, aiming to shrink model size without significantly reducing accuracy, thus improving usability and efficiency in practical applications.

Frontend compression methods include knowledge distillation, compact model structure design, and filter pruning. Knowledge distillation transfers knowledge from a complex model to a smaller one, allowing the small model to maintain high computational efficiency while achieving the performance of the complex model. For example, Hinton et al. [35] proposed knowledge distillation techniques that transfer teacher model knowledge through softened output probability distributions. Compact model structure design improves the convolution method of neural networks (e.g., using depthwise separable convolutions) to reduce computational parameters. MobileNet [36] is a successful example in this aspect. Filter pruning removes unimportant weight matrices to reduce model redundancy.

Backend compression methods include low-rank approximation and unrestricted pruning. Low-rank approximation reconstructs large weight matrices with several low-rank matrices, reducing storage and computational resource consumption. For example, Singular Value Decomposition (SVD)[37] is widely used for matrix decomposition to achieve compression. Unrestricted pruning includes unstructured pruning and structured pruning. Unstructured pruning removes individual weights.

In addition to these traditional compression methods, Han Song’s team proposed AutoML Model Compression (AMC) [38], utilizing reinforcement learning to automatically search for model compression strategies, enhancing the efficiency of deploying neural network models on mobile devices. AMC uses reinforcement learning to intelligently balance model size, speed, and accuracy, automatically generating optimal compression strategies more efficiently and effectively than manually crafted heuristic rules.

These model compression techniques improve computational and storage efficiency, allowing deep neural networks to be widely applied in resource-constrained environments.

Pruning techniques remove redundant parameters and connections in a model to enhance computational efficiency and reduce model size. Pruning techniques can be categorized into unstructured pruning, structured pruning, and hybrid pruning.

Unstructured pruning operates at a fine granularity, removing arbitrary ”redundant” parameters in the network. However, this method may result in irregular network structures that are difficult to accelerate effectively. LeCun proposed the Optimal Brain Damage (OBD) algorithm [39] in the late 1980s, using the second-order derivatives of the loss function to determine parameter importance. Hassibi et al. [40, 41] extended this with the Optimal Brain Surgeon (OBS) algorithm, not limited by OBD’s diagonal assumption, zeroing out less important weights and recalculating others to compensate for activation values, achieving better compression results. Srinivas et al. [42] proposed methods to remove dense connections in fully connected layers without relying on training data, significantly reducing computational complexity.

Structured pruning removes structural components based on predefined criteria, such as attention heads or layers.X-Pruner [43] utilizes explainable masks learned end-to-end, measuring each unit’s contribution to predicted target classes, and adaptively searches layer-wise thresholds to retain the most informative units while determining pruning rates.

Hybrid pruning combines unstructured and structured pruning methods, balancing their advantages for better performance optimization. For example, SPViT [44] developed a multi-head token selector based on dynamic attention for adaptive instance-level token selection, introducing a soft pruning technique that merges less important tokens into packet tokens rather than discarding them. ViT-Slim [45] introduced learnable and unified sparsity constraints with predefined factors to represent global importance within various dimensions.

These pruning techniques demonstrate significant effectiveness in various applications. In image classification tasks, structured pruning can significantly reduce convolutional neural networks’ computational costs while maintaining high classification accuracy. In natural language processing tasks, unstructured and hybrid pruning effectively reduce Transformer model complexity, enabling inference and training with lower resource consumption. By applying these pruning techniques, models maintain high performance while significantly reducing computational and storage costs, enhancing their usability and efficiency in practical applications.

Knowledge Distillation (KD) is a model compression technique that transfers knowledge from a complex model (called the teacher model) to a smaller model (called the student model). This allows the student model to maintain high computational efficiency while achieving the performance of the teacher model. Knowledge distillation was first proposed by Buciluǎ et al., who trained compressed models with pseudo-data classifiers to replicate the original classifier’s outputs[14]. KD can be divided into homomorphic KD and heteromorphic KD.

Homomorphic KD means the student and teacher models have similar or identical structures. In this approach, the student model learns by mimicking the teacher model’s outputs (e.g., logits, feature layer outputs). Common homomorphic KD methods include logit-level distillation, feature-level distillation, and module-level distillation. For instance, TinyViT[46] applies distillation during pre-training, storing logits from a large teacher model on hardware to achieve memory and computational efficiency when transferring knowledge to a smaller student Transformer. DeiT-Tiny[47] adopts patch-level distillation, training a small student model to match the pre-trained teacher model’s patch structure, then optimizing with decomposed manifold matching loss to reduce computational costs. Module-level methods like m2mKD[48] separate the teacher module from a pre-trained unified model, combining student modules with modular models, and using a shared meta-model for composition, enabling student modules to mimic teacher module behavior. Feature-level distillation methods like MiniViT[49] combine weights from consecutive Transformer blocks for cross-layer weight sharing, introducing transformations to enhance learning.

Heteromorphic KD refers to student and teacher models with different structures. In this approach, the student model learns by mimicking the teacher model’s outputs or intermediate features, despite different architectures. Heteromorphic KD enhances the student model’s adaptability, enabling it to learn useful information from the teacher model. Heteromorphic KD includes soft label distillation, where the student model trains by mimicking the teacher model’s soft label outputs.

KD transfers knowledge from complex models to smaller models, achieving model compression and acceleration. Both homomorphic and heteromorphic KD train by mimicking the teacher model’s outputs or features. These methods not only improve student model performance but also reduce computational and storage costs, enabling deep learning models to be widely applied in resource-constrained environments. Studies show that models processed by KD can perform well in resource-constrained environments such as mobile devices and embedded systems, further promoting deep learning technology’s wide deployment in practical applications.

Quantization techniques convert model parameters from high-precision floating-point numbers (e.g., 32-bit or 64-bit) to lower-precision formats (e.g., 8-bit or 16-bit), reducing computational and storage costs[50, 51]. For example, when training a cat-dog classification model on a laptop, its parameter size might be 64MB. Deploying it on an Arduino Uno using an ATmega328P microcontroller with 8-bit operations, quantizing the model can reduce the weight storage size to 1/8 of the original, with negligible accuracy impact (about 1-3%). This demonstrates quantization’s significant advantages in reducing storage needs and improving computational efficiency.

Weights are trainable parameters in neural networks, adjusted during training to minimize the model’s loss function, enabling the model to learn from data. Each layer’s weights transform input features to output features through matrix multiplication. Suppose the input vector is $\mathbf{x}$, weight matrix $\mathbf{W}$, and bias vector $\mathbf{b}$, then the neural network layer output can be represented as:

Quantization techniques convert model parameters from high-precision floating points to lower-precision formats, effectively reducing computational and storage costs. Quantization methods include post-training quantization, quantization-aware training, and hardware-aware quantization.

Post-Training Quantization (PTQ) quantizes model parameters after training completion. PTQ’s main advantage is simplicity and speed, not requiring adjustments to the training process. It typically includes weight and activation quantization. Quantization-Aware Training (QAT) considers quantization impacts during training, simulating quantization effects to help the model adapt to post-quantization performance. QAT typically includes weight and activation quantization. Hardware-Aware Quantization (HAQ) considers specific hardware architecture characteristics during quantization, optimizing model performance on specific hardware. HAQ not only considers scaling factors but also combines hardware features for optimization, such as adjusting scaling factors and quantization ranges to suit hardware characteristics.

The industry widely adopts INT8 quantization, replacing FP32 during inference while training still uses FP32. Many deep learning software such as TensorRT, TensorFlow, PyTorch, and MxNet have enabled or are enabling quantization. Quantization techniques enable deep learning models to be widely applied in resource-constrained environments while maintaining high computational performance and low storage requirements.

Synthetic data techniques generate data similar to real data but without containing real personal information, expanding training datasets to improve model generalization and robustness[52, 53]. In large model training, pure text synthesis is mostly done through other large models, while image synthesis mainly uses generative models. Synthesized data often needs to be validated through statistical methods to ensure conformity with real data distribution.

Statistical methods generate synthetic data by performing statistical analysis and modeling on real data, then using these models to generate synthetic data. For example, using probability distribution functions to simulate real data characteristics and distribution to generate synthetic data. Generative Adversarial Networks (GANs) are deep learning techniques used to generate realistic synthetic data. GANs consist of a generator and a discriminator, where the generator produces synthetic data, and the discriminator distinguishes between real and synthetic data. Through continuous adversarial training, the generator and discriminator compete, ultimately generating high-quality synthetic data. GANs have wide applications in medical imaging, facial recognition, and autonomous driving. Variational Autoencoders (VAEs) are generative models that learn data latent representations to generate synthetic data similar to real data distribution. VAEs are particularly suitable for image generation tasks, performing well in generating high-quality, realistic images. Sequence models generate synthetic data for sequence data (e.g., text, time series) through models such as Markov Chains, Recurrent Neural Networks (RNNs), and Variational Autoencoders (VAEs), modeling sequence features and dependencies to generate synthetic data.

Evaluation techniques for model architectures measure and compare the performance of different deep learning models to select the best architecture. Evaluation methods can be divided into manual and automatic evaluations.

Manual evaluation involves experts or users assessing model outputs, suitable for tasks with strong subjectivity, such as the quality of generated text or the realism of generated images. However, manual evaluation is inefficient, costly, and difficult to scale.

Automatic evaluation measures model performance by computing various performance metrics, offering high efficiency and repeatability. Common automatic evaluation platforms and tools include Prompt Flow in Microsoft Azure AI Studio, Weights Biases combined with LangChain, LangSmith[54] in LangChain, DeepEval in Confidence-ai, and TruEra[55]. These platforms and tools provide various evaluation methods, such as rule-based and model-based evaluations.

Rule-based evaluation methods use predefined rules and metrics (e.g., accuracy, precision, recall, F1 score, ROC-AUC curve) to assess model performance. For example, datasets like MMLU[56], TriviaQA[57], and HumanEval[58] are widely used to evaluate language model understanding and generation capabilities. Model-based evaluation methods use pre-trained referee models (e.g., GPT-4, Claude) or adversarial evaluation (e.g., LLM Peer-examination) to assess model performance. These methods comprehensively evaluate model performance on complex tasks and multimodal data.

Fine-tuning techniques involve further training pre-trained models on specific task datasets to enhance model performance in that task. Below are common fine-tuning techniques and their recent advancements:

LoRA (Low-Rank Adaptation)[59] is a low-rank adaptation technique that adds low-rank matrices to pre-trained models for fine-tuning, reducing computational and storage costs while maintaining performance. QLoRA[60] is an improved version that further optimizes the fine-tuning process through quantization techniques. Retrieval-Augmented Generation (RAG)[61] combines information retrieval and generative models, enhancing generative model performance by retrieving relevant information from external data sources. The LangChain[62] library provides various tools allowing large models to access real-time information from sources like Google Search, vector databases, or knowledge graphs, further improving RAG effectiveness. LlamaIndex (GPT Index)[63, 64] is an integrated data framework designed to enhance large language models (LLMs) by enabling the use of private or custom data. LlamaIndex provides data connectors, indexing and graph-building mechanisms, and advanced retrieval and query interfaces, simplifying data integration and information retrieval processes.

By applying these fine-tuning techniques appropriately, pre-trained model knowledge can be fully utilized, improving performance in new tasks while reducing training time and computational resource consumption.

In the field of multimodal large models, the Transformer architecture is widely used for its excellent performance and flexibility. However, as model size and application demands increase, the Transformer architecture faces challenges in computational complexity and memory bottlenecks. To address these challenges, researchers have proposed various optimization strategies and alternative architectures to enhance model efficiency and scalability. Besides Transformers, most other challenger architectures originate from recurrent neural networks (RNNs), including Gated Convolution, Temporal Convolutional Networks (TCN), RWKV, Mamba, and S4, which replace attention with recurrent structures. This approach uses fixed memory to remember previous information, although it can remember a certain length, achieving longer lengths is challenging. Another approach is improving Transformers, such as linear attention improvements mentioned earlier. Representative models include Mega, Yan, and JEPA. We will introduce some representative approaches among them.

The RWKV model[22] uses linear attention mechanisms, allowing the model to parallelize computations during training and maintain constant computational and memory complexity during inference. The RWKV model consists of stacked residual blocks, each containing time-mixing and channel-mixing sub-blocks, using a recurrent structure to leverage past information. The authors trained RWKV models with sizes ranging from 169 million to 14 billion parameters, making it the largest dense RNN trained to date. Experimental results show that RWKV performs comparably to Transformers of similar size, indicating future work can utilize this architecture to create more efficient models. However, RWKV models have some limitations, such as linear attention potentially limiting performance on tasks requiring long-term dependencies.

The Mega model[23] introduces sparse attention mechanisms, zeroing out most elements in the attention matrix and retaining only a few important attention values. This method significantly reduces computational load and memory usage while maintaining predictive performance. Similar to Longformer and Sparse Transformer, the Mega model has unique optimizations in sparse strategies and implementations. By using sparse attention mechanisms, the Mega model greatly reduces computational complexity and memory usage, making it more efficient in handling long-sequence tasks.

JEPA (Joint Embedding Predictive Architecture)[24] is a novel machine learning model designed to optimize complex tasks and large-scale problem handling through hierarchical decision-making and control methods. The core idea is to decompose problems into multiple layers, each handling subtasks at different abstraction levels, simplifying the overall problem-solving process. The concept and research of JEPA are mainly proposed by Yann LeCun’s team at Meta, aiming to overcome the limitations of current large language models (LLMs) in handling complex tasks. A representative method is I-JEPA, a non-generative self-supervised learning method that learns highly semantic image representations by predicting representations of different target blocks in the same image from a single context block. This novel architecture combines the strengths of RNNs and Transformers while reducing their limitations.

Mamba and Mamba 2[25, 26] are key directions of improvement. Mamba is an improvement of SSM. State Space Models (SSM) describe dynamic systems and are widely used in control theory, signal processing, and statistical modeling. SSM uses state variables to represent the system’s internal state, described by state and output equations. The state equation is:

where $h(t)$ is the state vector at time $t$, $A$ is the state transition matrix, $x(t)$ is the input vector at time $t$, $B$ is the input matrix, and $w(t)$ is process noise, usually assumed to be zero-mean Gaussian white noise.

The output equation is:

where $y(t)$ is the output vector at time $t$, $C$ is the output matrix, $D$ is the direct transmission matrix, and $v(t)$ is measurement noise, usually assumed to be zero-mean Gaussian white noise.

Mamba is a Selective State Space Model (SSSM) based on SSM improvements. Figure 4 shows the architectural differences between SSM and Transformer when handling multidimensional input data. SSM processes each dimension independently, with high parallel computing capabilities and linear computational complexity. In contrast, Transformers capture global dependencies through multi-head attention mechanisms but have higher computational complexity.

H3 architecture[21], a foundational design for homogenized architecture, improves the initial SSM structure, addressing SSM’s challenge of remembering long-term data. As shown in Figure 5, researchers merged the previous SSM architectural design H3 with Gated MLP blocks into one block, selectively processing input information (Selection Mechanism), simplifying the deep sequential model architecture, forming a simple, homogeneous architecture with selective state spaces (Mamba). Like structured SSM, selective SSM is an independent sequence transformation, flexibly integrating into neural networks.

Mamba-2 proposes the State Space Duality (SSD) framework. Based on this, researchers designed the new Mamba-2 architecture, with its core layer being an improved selective SSM. Researchers mixed 4-6 attention layers with Mamba-2 layers, outperforming Transformer++ and pure Mamba-2, indicating attention and SSM are complementary.

One main goal of Mamba-2 is to accelerate SSM using tensor cores. In Mamba-2’s SSD structure, parallel projections are realized, breaking through SSM’s sequential computation limitations to achieve parallel computation. In actual architecture changes, some SSM parameters being internal activation functions (states) rather than layer input functions limit parallel computation and training speed. In Mamba-2, all SSM parameters are layer input functions, easily applying tensor parallelism to input projections.

Researchers trained a series of Mamba-2 models on the Pile dataset, showing Mamba-2 matches or exceeds Mamba and open-source Transformers in standard downstream evaluations. For instance, a 2.7B parameter Mamba-2 trained on 300 billion tokens in the Pile dataset outperforms the 2.8B parameter Mamba and Pythia, as well as the 6.9B parameter Pythia.

Multimodal models have demonstrated significant potential and application prospects in processing and understanding data from different modalities. By analyzing existing multimodal models, it is evident that they have made remarkable progress in generating images, videos, audio, and 3D models. These models achieve cross-modal generation and transformation by handling data from various modalities such as text, images, videos, or audio.

For example, the DALL-E series models (including DALL-E[98], DALL-E 2[82], DALL-E 3[99]) developed by OpenAI since 2021 have evolved to generate high-quality images based on textual descriptions. DALL-E models leverage the power of large-scale datasets and transformer architectures to understand and generate highly detailed and creative images from textual inputs. Each iteration has improved on the previous, with DALL-E 3 incorporating more advanced techniques to produce even more coherent and higher-quality images.

Midjourney[100], launched in 2022, focuses on high-quality artistic image generation and is widely used in creative design. This model emphasizes generating visually appealing and artistically valuable images, making it popular among artists and designers.

Google’s Imagen and Imagen 2[79], introduced in 2022 and 2023, respectively, have further enhanced image generation capabilities by including editing features. Imagen models allow users to make modifications to generated images, making the process of creating and refining images more interactive and user-friendly.

Stability AI’s Stable Diffusion[78], open-sourced in 2022, has propelled community research and application in image generation. The open-source nature of Stable Diffusion has encouraged a wide range of research and development activities, contributing to the rapid advancement of image generation technologies.

The field of video generation has also seen significant breakthroughs. OpenAI and PKU-YuanGroup released Sora and Open-Sora in 2023, capable of generating highly realistic videos. These models extend the capabilities of image generation to the temporal domain, allowing for the creation of dynamic and lifelike video content.

AILab-CVC’s VideoCrafter and VideoCrafter2[101], released in 2023 and 2024 respectively, further optimized video generation and editing functionalities. These models integrate advanced video editing capabilities, making it easier to create and modify video content. Google’s AudioLM[102], launched in 2022, excels in audio generation, producing natural sounds, music, and human speech. This model showcases the potential of multimodal generative models in creating high-quality audio content from textual descriptions. Fudan University’s SpeechGPT[103], introduced in 2023, combines speech recognition and generation technologies, enhancing the naturalness and accuracy of voice interactions. SpeechGPT integrates advanced language models with speech technologies to provide more fluent and accurate voice interactions.

Multimodal foundation models focus on understanding and aligning multimodal data, enabling more complex reasoning and task execution. DeepMind’s Flamingo, launched in 2022, understands image content and answers related questions, excelling in visual question answering tasks. InstructBLIP and Blip-2, introduced between 2022 and 2023, improved visual question answering performance through efficient visual-language pre-training.

DeepMind’s Flamingo[104], launched in 2022, understands image content and answers related questions, excelling in visual question answering tasks. This model is designed to bridge the gap between visual inputs and textual queries, providing accurate and contextually relevant answers.

InstructBLIP and Blip-2[105, 106], introduced between 2022 and 2023, improved visual question answering performance through efficient visual-language pre-training. These models leverage large-scale pre-training to enhance their ability to understand and respond to visual queries, making them more effective in various visual question answering scenarios.

Alibaba DAMO Academy’s video understanding models, VideoLlama and VideoLlama 2[107, 108], released in 2023 and 2024 respectively, can understand video content and perform tasks such as video subtitle generation and video-audio understanding. These models are designed to process and comprehend complex video data, enabling more sophisticated video analysis and understanding tasks.

Multimodal models like MiniGPT4[109], mPLUG-Owl[110], and LlaVA[67], developed by KAUST and Alibaba DAMO Academy in 2023, support input and output of text and images. They achieve comprehensive text and image processing through efficient multimodal alignment techniques. These models are designed to handle a variety of multimodal tasks, providing robust performance in text-image alignment and processing.

Meituan and Zhejiang University developed MobileVLM and MobileVLMv2[111, 112] in 2023 and 2024, optimizing model performance and application scenarios for mobile platforms, further enhancing multimodal task execution capabilities. These models are specifically designed to be efficient and effective on mobile devices, enabling advanced multimodal functionalities on resource-constrained platforms.

3D generation is an essential area in multimodal generation, leading towards world simulators through models that generate realistic 3D models and incorporate rule constraints in the generation process to create highly realistic and controllable virtual environments, akin to the metaverse.

3D generation techniques mainly include explicit representation, implicit representation, and hybrid representation. Explicit representation includes point clouds and meshes, which generate 3D models by precisely describing the geometry of objects. Implicit representations, such as Neural Radiance Fields (NeRF)[113] and implicit surfaces, generate high-quality 3D content by learning latent representations of data. Hybrid representations combine explicit and implicit features, retaining geometric details while offering flexible representation capabilities.

Specific generation methods include Generative Adversarial Networks (GANs), diffusion models, autoregressive models, Variational Autoencoders (VAEs), and normalizing flows. These methods generate realistic 3D data through various mechanisms. For example, GANs generate high-quality 3D models through adversarial training between generators and discriminators; diffusion models generate new samples by simulating the diffusion process of data; autoregressive models generate 3D objects by progressively predicting the conditional probability of each element; VAEs generate data by learning latent representations of input data; normalizing flows use a series of reversible transformations to map simple distributions to data distributions for data generation[114].

Optimization-based generation methods use optimization techniques to generate 3D models at runtime, often combining pre-trained networks to optimize 3D models based on user-specified prompts (such as text or images). For example, text-to-3D techniques use textual prompts to guide 3D content generation; image-to-3D techniques reconstruct 3D models from specified images, preserving image appearance and optimizing 3D content geometry. Procedural generation uses predefined rules, parameters, and mathematical functions to create 3D models and textures, including fractal geometry, L-systems, noise functions, and cellular automata[115, 116].

Generative novel view synthesis uses generative techniques to predict new views from a single input image, generating new content based on conditional 3D information. Transformer-based methods use multi-head attention mechanisms to gather information from different positions for novel view synthesis; GAN-based methods use 3D point clouds as representations, synthesizing missing regions and generating output images through GANs. These methods have their advantages and application scenarios, and researchers can choose suitable 3D generation techniques based on specific needs[117, 118].

Despite significant improvements in 3D generation quality and diversity, current challenges include evaluation, dataset size and quality, representation flexibility, and controllability. Multimodal large models require deeper networks and larger datasets for pre-training. Multimodal large models are often pre-trained on vision and language modalities, and future expansions can include more modalities such as images, text, audio, time, thermal images, etc. Large-scale pre-trained models based on multiple modalities have broader application potential.

Another approach to achieving world simulators is through embodied intelligence models. Current multimodal models cover daily information media such as images, text, and audio. However, developing embodied intelligent robots requires expanding these modal information to include coordinate systems, point clouds, and depth, which are crucial for robots to understand and operate in the real world. Integrating these additional modal information into multimodal models can achieve preliminary embodied intelligent robots[119, 120].

Embodied intelligence involves robots perceiving, understanding, and acting in the physical world. To achieve this, robots need to process and understand data from multiple sensors, such as cameras, LiDAR, and depth sensors[121, 122]. These sensors provide information, including coordinate systems, point clouds, and depth maps, enabling robots to construct and understand detailed 3D representations of their surroundings. With this modal information, robots can navigate, recognize objects, and perform tasks in the real world.

By deploying robots in real life, sufficient multimodal information can be collected to achieve comprehensive data collection of the real world. The performance of embodied intelligence can be further enhanced by combining sensor information from multiple sensors on embodied intelligence and edge devices collecting sensor information. With sufficient comprehensive modal information, not only can the performance of embodied intelligence be further improved, but it can also be used to design more realistic world simulators. Such a world simulator can provide robots with a virtual training environment, allowing them to learn and optimize under safe and controlled conditions. Through continuous iteration and optimization, highly intelligent and autonomous embodied intelligent robots can eventually be realized.

In the process of constructing world simulators, incorporating more external rule systems is a crucial approach. Humans rely on mathematical, physical, chemical, and biological tools in the objective world, using a series of theorems to derive and predict the outcomes of events that have not yet occurred. For example, when we kick a ball, it will fly in an arc. These predictions based on physical laws help us understand and operate the real world.

Similarly, rule systems can help models achieve state memory and feedback. Suppose a flood breaks the dam; the model needs to infer the subsequent flood state based on rules. These rules stem from human common sense and theorem libraries, summarized from long-term practice and experience. By injecting these conclusions into the model, the model can infer reasonable results with fewer data.

In constructing multimodal large models, integrating external rule systems can significantly enhance model understanding and reasoning capabilities. For example, using mathematical theorems, the model can accurately calculate the trajectory of objects; using physical laws, the model can predict complex environmental changes; using biological knowledge, the model can simulate dynamic changes in ecosystems. These rule systems provide the model with a framework, allowing it to more accurately simulate the real world.

In practical applications, embodied intelligent robots can benefit from these rule systems. When robots collect vast amounts of multimodal data in real life, these data will combine with injected rule systems to enhance robots’ prediction and decision-making abilities. For example, when a robot detects rising water levels, it can predict potential flood ranges and impacts based on physical and geographical knowledge and take corresponding actions.

By incorporating these external rule systems, multimodal large models can excel in various application scenarios and achieve more complex and detailed tasks. This approach not only enhances model intelligence but also provides a more solid foundation for future development.