DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models

The overall pipeline of DriveVLM is illustrated in Figure 1. A sequence of images is processed by a Vision Language Model (VLM) to perform a special chain-of-thought (CoT) [49] reasoning to derive the driving planning results. The architecture of DriveVLM involves a vision transformer encoder [50] and a Large Language Model (LLM). The vision encoder produces image tokens; then an attention-based extractor aligns these tokens with the LLM. The reasoning process can be divided into three modules: scene description (Section 3.2), scene analysis (Section 3.3), and hierarchical planning (Section 3.4).

For real-world deployment, we propose a hybrid system, DriveVLM-Dual, in Section 3.5, which combines DriveVLM and the traditional autonomous driving pipeline, leveraging the strengths of both approaches.

The scene description module identifies driving environment description and critical objects.

Environment Description. Driving environments, such as weather and road conditions, have a non-negligible impact on driving difficulty. Therefore, the model is first prompted to output a linguistic description $E$ of the driving environment, including several conditions: $E=\{E_{\text{weather}},E_{\text{time}},E_{\text{road}},E_{\text{lane}}\},$ each representing a crucial aspect of the driving environment. The weather component, $E_{\text{weather}}$, spans conditions from sunny to snowy, affecting visibility and traction. The time component, $E_{\text{time}}$, distinguishes between daytime and nighttime, impacting driving strategies due to visibility changes. Road types, $E_{\text{road}}$, such as urban or highway, introduce different challenges, while lane conditions, $E_{\text{lane}}$, focus on current lane positioning and possible maneuvers, crucial for safe driving decisions.

Critical Object Identification. In addition to environmental conditions, various objects in driving scenarios significantly influence driving behaviors. Unlike traditional autonomous driving perception modules, which detect all objects within a specific range, we solely focus on identifying critical objects that are most likely to influence the current scenario, inspired by human cognitive processes during driving. Each critical object, denoted as $O_{c}$, contains two attributes: the object category $c$ and its approximate bounding box coordinates $b(x1,y1,x2,y2)$ on the image. The category and coordinates are mapped to their corresponding language $token\_id$ in the language modality, enabling seamless integration into the following modules. Moreover, taking advantage of the pre-trained vision encoder, DriveVLM can identify long-tail critical objects that may elude typical 3D object detectors, such as road debris or unusual animals.

In the traditional autonomous driving pipeline, the prediction module typically concentrates on forecasting the future trajectories of objects. The emergence of advanced vision-language models has provided us with the ability to perform a more comprehensive analysis of the current scene. The scene-level analysis summarizes all the critical objects together with the environmental description. This summary gives a comprehensive understanding of the scene, and is fed into the following planning module.

Critical Object Analysis. DriveVLM characterizes critical objects in three aspects: static attributes $C_{s}$, motion states $C_{m}$, and particular behaviors $C_{b}$. Static attributes $C_{s}$ describe inherent properties of objects, such as a roadside billboard’s visual cues or a truck’s oversized cargo, which are critical in preempting and navigating potential hazards. Motion states $C_{m}$ describe an object’s dynamics over a period, including position, direction, and action—characteristics that are vital in predicting the object’s future trajectory and potential interactions with the ego vehicle. Particular behaviors $C_{b}$ refer to special actions or gestures of an object that could directly influence the ego vehicle’s next driving decisions. We do not require the model to analyze all three characteristics for all objects. In practice, only one or two characteristics apply to a critical object. Upon analyzing these characteristics, DriveVLM then predicts the potential influence $I$ of each critical object on the ego vehicle.

The scene-level summary is then combined with the route, ego pose and velocity to form a prompt for planning. Finally, DriveVLM progressively generates driving plans, in three stages: meta-actions, decision description, and trajectory waypoints.

Meta-actions $\boldsymbol{A}$. A meta-action, denoted as $a_{i}$, represents a short-term decision of the driving strategy. These actions fall into 17 categories, including but not limited to acceleration, deceleration, turning left, changing lanes, minor positional adjustments, and waiting. To plan the ego vehicle’s future maneuver over a certain period, we generate a sequence of meta-actions.

Decision Description $\boldsymbol{D}$. Decision description $D$ articulates the more fine-grained driving strategy the ego vehicle should adopt. It contains three elements: Action $\mathcal{A}$, Subject $\mathcal{S}$, and Duration $\mathcal{D}$. Action pertains to meta actions such as ‘turn’, ‘wait’, or ‘accelerate’. Subject refers to the interacting object, such as a pedestrian, a traffic signal, or a specific lane. Duration indicates the temporal aspect of the action, specifying how long it should be carried out or when it should start.

Trajectory Waypoints $\boldsymbol{W}$. Upon establishing the decision description ${D}$, our next phase involves the generation of corresponding trajectory waypoints. These waypoints, denoted by $W=\{w_{1},w_{2},...,w_{n}\}$, $w_{i}=(x_{i},y_{i})$, depict the vehicle’s path over a certain future period with predetermined intervals $\Delta t$. We map these numerical waypoints into language tokens for auto-regressive generation.

To mitigate the challenges of high latency and imprecise spatial and motion understanding in VLMs, we propose DriveVLM-Dual, a collaboration between DriveVLM and the traditional autonomous driving system. This novel approach involves two key strategies: incorporating 3D perception for critical object analysis, and high-frequency trajectory refinement.

Integrating 3D Perception. We represent objects detected by a 3D detector as $O_{\text{3D}}=\{c_{\text{3D}}^{i},b_{\text{3D}}^{i}\}$, where $b_{\text{3D}}^{i}$ denotes the $i$-th bounding box and $c_{\text{3D}}^{i}$ denotes its category. These 3D bounding boxes are then back-projected onto 2D images to derive corresponding 2D bounding boxes $b_{\text{2D}}^{i}$. We conduct IoU matching between these 2D bounding boxes $b_{\text{2D}}^{i}$ and $b_{c}^{j}$. $b_{c}^{j}$ are the bounding boxes of previously identified critical objects $O_{\text{critical}}=\{c_{c}^{j},b_{c}^{j}\}$. We classify critical objects that meet a certain approximate IoU threshold and belong to the same category as matched critical objects $O_{c}^{\text{matched}}$, defined as

$\displaystyle O_{c}^{\text{matched}}=\{c_{c}^{j},b_{c}^{j}\},\quad$

$\displaystyle\text{if }c_{c}^{j}=c_{\text{2D}}^{i}\text{ and }\text{aIoU}(b_{c}^{j},b_{\text{2D}}^{i})>\tau,\text{ where aIoU}(b_{c}^{j},b_{\text{2D}}^{i})=\frac{S_{b_{c}^{j}\cap b_{\text{2D}}^{i}}}{S_{b_{\text{2D}}^{i}}},$

Those critical objects without a corresponding match in the 3D data are noted as $O_{c}^{\text{unmatched}}$.

In the scene analysis module, for $O_{c}^{\text{matched}}$, the center coordinates, orientations, and historical trajectories of the corresponding 3D objects are used as language prompts for the model, assisting in object analysis. Conversely, for $O_{c}^{\text{unmatched}}$, analysis relies solely on the language tokens derived from the image. This design enables DriveVLM-Dual to understand the locations and motions of critical objects more accurately, enhancing the overall performance.

High-frequency Trajectory Refinement. To achieve real-time, high-frequency inference capabilities, we integrate it with a conventional planner to form a slow-fast dual system, combining the advanced capabilities of DriveVLM with the efficiency of traditional planning methods. After obtaining a trajectory from DriveVLM at low frequency, denoted as $W_{\text{slow}}$, we take it as a reference trajectory for a classical planner for high-frequency trajectory refinement. In the case of an optimization-based planner, $W_{\text{slow}}$ serves as the initial solution for the optimization solver. For a neural network-based planner, $W_{\text{slow}}$ is used as an input query, combined with additional input features $f$, and then decoded into a new planning trajectory denoted as $W_{\text{fast}}$. The formulation of this process can be described as:

$\displaystyle W_{\text{fast}}=\text{Planner}([W_{\text{slow}},f]).$

(1)

This refinement step ensures that the trajectory produced by DriveVLM-Dual (1) achieves higher trajectory quality, and (2) meets real-time requirements. In practice, the two branches operate asynchronously in a slow-fast manner, where the planner module in the traditional autonomous driving branch can selectively receive trajectory from the VLM branch as additional input.

The Scene Understanding for Planning task is defined as follows. The input comprises multi-view videos $\mathcal{V}$ from surrounding cameras and optionally 3D perception results $\mathcal{P}$ from a perception module. The output includes the following components:

1. 1.

   Scene Description $E$: Composed of weather condition $E_{\text{weather}}$, time $E_{\text{time}}$, road condition $E_{\text{road}}$, and lane conditions $E_{\text{lane}}$.

2. 2.

   Scene Analysis $S$: Including object-level analysis and scene-level summary $S$.

3. 3.

   Meta Actions $A$: A sequence of actions representing task-level maneuvers.

4. 4.

   Decision Description $D$: A detailed account of the driving decisions.

5. 5.

   Trajectory Waypoints $W$: The waypoints outlining the planned trajectory of the ego vehicle.

To comprehensively evaluate a model’s performance, we care about its interpretation of the driving scene and the decisions made. Therefore, our evaluation has two aspects: scene description/analysis evaluation and meta-action evaluation.

Scene Description/Analysis Evaluation. Given the subjective nature of human evaluation in scene description, we adopt a structured approach using a pre-trained LLM. This method entails comparing the generated scene description with a human-annotated ground truth description. The ground truth description encompasses structured data such as environmental conditions, navigation, lane information, and critical events with specific objects, verbs, and their influences. The LLM assesses and scores the generated descriptions based on their consistency with the ground truth.

Meta-action Evaluation. Meta-actions are a predefined set of decision-making options. A driving decision is formulated as a sequence of meta-actions. Our evaluation method employs a dynamic programming algorithm to compare the model-generated sequences with a manually annotated ground truth sequence. The evaluation should also weigh the relative importance of various meta-actions, designating some as ‘conservative actions’ with a lower impact on the sequence’s overall context. To increase robustness, we first use the LLM to generate semantically equivalent alternatives to the ground truth sequence to enhance robustness. The sequence with the highest similarity to these alternatives calculates the final driving decision score. More details of the proposed metric are available in the Appendix B.

We propose a comprehensive data mining and annotation pipeline, shown in Figure 3, to construct a Scene Understanding for Planning (SUP-AD) Dataset for the proposed task. Specifically, we perform long-tail object mining and challenging scenario mining from a large database to collect samples, then we select a keyframe from each sample and further perform scene annotation. Dataset statistics are available in the Appendix A.

Long-tail Object Mining. According to real-world road object distribution, we first define a list of long-tail object categories, such as weird-shaped vehicles, road debris, and animals crossing the road. Next, we mine these long-tail scenarios using a CLIP-based search engine, capable of mining driving data using language queries from a large collection of logs. Following that, we perform a manual inspection to filter out scenes inconsistent with the specified categories.

Challenging Scenario Mining. In addition to long-tail objects, we are also interested in challenging driving scenarios, where the driving strategy of the ego vehicle needs to be adapted according to the changing driving conditions. These scenarios are mined according to the variance of the recorded driving maneuvers.

Keyframe Selection. Each scene is a video clip, it is essential to identify the ‘keyframe’ to annotate. In most challenging scenarios, a keyframe is the moment before a significant change in speed or direction is required. We select this keyframe 0.5s to 1s earlier than the actual maneuver, based on comprehensive testing, to guarantee an optimal reaction time for decision-making. For scenes that do not involve changes in driving behavior, we select a frame that is relevant to the current driving scenario as the keyframe.

Scene Annotation. We employ a group of annotators to perform the scene annotation, including scene description, scene analysis, and planning, except for waypoints, which can be auto-labeled from the vehicle’s IMU recordings. To facilitate scene annotation, we make a video annotation tool with the following features: (1) the annotators can slide the progress bar back and forth to replay any part of a video; (2) while annotating a keyframe, the annotator can draw bounding boxes on the image together with language descriptions; (3) annotators can select from a list of action and decision candidates while annotating driving plans. Each annotation is meticulously verified by 3 annotators for accuracy and consistency, ensuring a reliable dataset for model training. Figure 2 illustrates a sample scenario with detailed annotations.

We test DriveVLM and DriveVLM-Dual on our proposed SUP-AD dataset and nuScenes dataset [44].

SUP-AD Dataset. The SUP-AD dataset is a dataset built by our proposed data mining and annotation pipeline. It is divided into train, validation, and test splits with a ratio of $7.5:1:1.5$. We train models on the training split and use our proposed scene description and meta-action metrics to evaluate model performance on the test split. We also employ co-tuning with additional datasets to ensure the generalization of the LLM is not compromised.

nuScenes Dataset. The nuScenes dataset is a large-scale driving dataset of urban scenarios with 1000 scenes, where each scene lasts about 20 seconds. Following previous works [34, 51], we adopt Displacement Error (DE) and Collision Rate (CR) as metrics to evaluate models’ performance.

SUP-AD. We present the performance of our proposed DriveVLM with several large vision-language models and compare them with GPT-4V, as shown in Table 1. DriveVLM, utilizing Qwen-VL as its backbone, achieves the best performance due to its strong capabilities in question answering and flexible interaction compared to the other open-source VLMs. Although GPT-4V exhibits robust capabilities in vision and language processing, its inability to undergo fine-tuning, restricting it solely to in-context learning, often results in the generation of extraneous information during scene description tasks. Under our evaluation metric, the additional information is frequently classified as hallucination, consequently leading to lower scores.

nuScenes. As shown in Table 2, DriveVLM-Dual achieves state-of-the-art performance on the nuScenes planning task when cooperating with VAD. It demonstrates that our method, although tailored for understanding complex scenes, also excels in ordinary scenarios.

Model Design. To better understand the significance of our designed modules in DriveVLM, we conduct ablations on different combinations of modules, as shown in Table 3. The inclusion of critical object analysis enables our model to identify and prioritize important elements in the driving environment, enhancing the decision-making accuracy for safer navigation. Integrating 3D perception data, our model gains a refined understanding of the surroundings and achieves precise predictions.

Traditional AD Pipeline. To demonstrate the generalization of our dual system design, we test DriveVLM-Dual with different traditional autonomous driving pipelines on the validation set of nuScenes. As illustrated in Table 4, our proposed DriveVLM-Dual adapts well to different traditional AD pipelines. While a standalone MLP method shows a notable performance gap compared to VAD, both variants of DriveVLM-Dual achieve nearly identical performance, underscoring the efficacy and robustness of our dual system design.

Qualitative results of DriveVLM are shown in Figure 4. In Figure 4(a), DriveVLM accurately predicts the current scene conditions and incorporates well-considered planning decisions regarding the cyclist approaching us. In Figure 4(b), DriveVLM effectively comprehends the gesture of the traffic police ahead, signaling the ego vehicle to proceed, and also considers the person riding a tricycle on the right side, thereby making sensible driving decisions. These qualitative results demonstrate our model’s exceptional ability to understand complex scenarios and make suitable driving plans. More visualization of our model’s output is shown in the Appendix C.