NVlabs/OmniDrive on GitHub: Python, Models, Vision models, Domain-specific models, 3D, Autonomous vehicles, Planning, Reasoning.

OmniDrive: LLM-Agent for Autonomous Driving with 3D Perception, Reasoning and Planning

https://github.com/NVlabs/OmniDrive/assets/74858581/f64987a0-b890-416d-90c1-e0daaeb542d6

We present OmniDrive, a holistic Drive LLM-Agent framework for end-to-end autonomous driving. Our main contributions involve novel solutions in both model (OmniDrive-Agent) and benchmark (OmniDrive-nuScenes). The former features a novel 3D multimodal LLM design that uses sparse queries to lift and compress visual representations into 3D. The latter is constituted of comprehensive VQA tasks for reasoning and planning, including scene description, traffic regulation, 3D grounding, counterfactual reasoning, decision making and planning.

News

[2026/02/06] Added TensorRT support with TensorRT-Edge-LLM on NVIDIA DRIVE AGX Thor. [Link]
[2025/04/16] Added TensorRT support with TensorRT-LLM on X86_64. [Link]
[2025/02/26] OmniDrive is accepted to CVPR 2025.
[2024/07/18] OmniDrive-nuScenes model release. [HF]
[2024/05/02] OmniDrive-nuScenes dataset release. [Data]
[2024/05/02] Technical report release. [arXiv]

Getting Started

Please follow Environment Setup step by step.

Currently Supported Features

[x] OmnDrive Training Framework
[x] OmnDrive Dataset
[x] OmnDrive Checkpoint
[x] Evaluation
[x] Data Generation
[x] TensorRT Inference on X86_64 with TensorRT-LLM
[x] TensorRT Inference on NVIDIA DRIVE AGX Thor with TensorRT-Edge-LLM
[x] Tiny LLM

Visual Results

Joint End-to-end Planning and Reasoning

Interactive Conversation with Ego Vehicle

Counterfactual Reasoning of Planning Behaviors

Citation

If this work is helpful for your research, please consider citing:

@inproceedings{wang2025omnidrive,
  title={{OmniDrive}: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning},
  author={Shihao Wang and Zhiding Yu and Xiaohui Jiang and Shiyi Lan and Min Shi and Nadine Chang and Jan Kautz and Ying Li and Jose M. Alvarez},
  booktitle={CVPR},
  year={2025}
}

Acknowledgement

The team would like to give special thanks to the NVIDIA TSE Team, including Le An, Chengzhe Xu, Yuchao Jin, and Josh Park, for their exceptional work on the TensorRT deployment of OmniDrive.