OpenDriveLab/DriveAGI — stars, momentum and ranking

[CVPR 2024 Highlight] GenAD: Generalized Predictive Model for Autonomous Driving

[!IMPORTANT] 🌟 Stay up to date at opendrivelab.com!

NEWS
⭐ GenAD: OpenDV Dataset (CVPR 2024 Hightlight)
Vista (NeurIPS 2024)
DriveLM (ECCV 2024 Oral)
DriveData Survey (SCIENTIA SINICA Informationis 2024)
OpenScene
OpenLane-V2 Update

NEWS

[ NEW❗️] 2024/09/08 We released a mini version of OpenDV-YouTube, containing 25 hours of driving videos. Feel free to try the mini subset by following instructions at OpenDV-mini!

2024/05/28 We released our latest research, Vista, a generalizable driving world model. It's capable of predicting high-fidelity and long-horizon futures, executing multi-modal actions, and serving as a generalizable reward function to assess driving behaviors.

2024/03/24 OpenDV-YouTube Update: Full suite of toolkits for OpenDV-YouTube is now available, including data downloading and processing scripts, as well as language annotations. Please refer to OpenDV-YouTube.

2024/03/15 We released the complete video list of OpenDV-YouTube, a large-scale driving video dataset, for GenAD project. Data downloading and processing script, as well as language annotations, will be released next week. Stay tuned.

2024/01/24 We are excited to announce some update to our survey and would like to thank John Lambert, Klemens Esterle from the public community for their advice to improve the manuscript.

GenAD: OpenDV Dataset

opendv

Examples of real-world driving scenarios in the OpenDV dataset, including urban, highway, rural scenes, etc.

⭐ Generalized Predictive Model for Autonomous Driving (CVPR 2024, Highlight)

Paper | Video | Poster | Slides

🎦 The Largest Driving Video dataset to date, containing more than 1700 hours of real-world driving videos and being 300 times larger than the widely used nuScenes dataset.

Complete video list (under YouTube license): OpenDV Videos.
- The downloaded raw videos (mostly 1080P) consume about 3 TB storage space. However, these hour-long videos cannot be directly applied for model training as they are extremely memory consuming.
- Therefore, we preprocess them into conseductive images which are more flexible and efficient to load during training. Processed images consumes about 24 TB storage space in total.
- It's recommended to set up your experiments on a small subset, say 1/20 of the whole dataset. An official mini subset is also provided and you can refer to OpenDV-mini for details. After stablizing the training, you can then apply your method on the whole dataset and hope for the best 🤞.
[ New❗️] Mini subset: OpenDV-mini.
- A mini version of OpenDV-YouTube. The raw videos consume about 44 GB of storage space and the processed images will consume about 390 GB of storage space.
Step-by-step instruction for data preparation: OpenDV-YouTube.
Language annotation for OpenDV-YouTube: OpenDV-YouTube-Language.

Quick facts:

Task: large-scale video prediction for driving scenes.
Data source: YouTube, with careful collection and filtering process.
Diversity Highlights: 1700 hours of driving videos, covering more than 244 cities in 40 countries.
Related work: GenAD Accepted at CVPR 2024, Highlight
Note: Annotations for other public datasets in OpenDV-2K will not be released since we randomly sampled a subset of them in training, which are incomplete and hard to trace back to their origins (i.e., file name). Nevertheless, it's easy to reproduce the collection and annotation process on your own following our paper.

@inproceedings{yang2024genad,
  title={Generalized Predictive Model for Autonomous Driving},
  author={Jiazhi Yang and Shenyuan Gao and Yihang Qiu and Li Chen and Tianyu Li and Bo Dai and Kashyap Chitta and Penghao Wu and Jia Zeng and Ping Luo and Jun Zhang and Andreas Geiger and Yu Qiao and Hongyang Li},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2024}
}

Vista

Simulated futures in a wide range of driving scenarios by Vista. Best viewed on demo page.

🌏 A Generalizable Driving World Model with High Fidelity and Versatile Controllability (NeurIPS 2024)

Quick facts:

Introducing the world's first generalizable driving world model.
Task: High-fidelity, action-conditioned, and long-horizon future prediction for driving scenes in the wild.
Dataset: OpenDV-YouTube, nuScenes
Code and model: https://github.com/OpenDriveLab/Vista
Video Demo: https://vista-demo.github.io
Related work: Vista, GenAD

@inproceedings{gao2024vista,
 title={Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability}, 
 author={Shenyuan Gao and Jiazhi Yang and Li Chen and Kashyap Chitta and Yihang Qiu and Andreas Geiger and Jun Zhang and Hongyang Li},
 booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
 year={2024}
}

@inproceedings{yang2024genad,
  title={{Generalized Predictive Model for Autonomous Driving}},
  author={Jiazhi Yang and Shenyuan Gao and Yihang Qiu and Li Chen and Tianyu Li and Bo Dai and Kashyap Chitta and Penghao Wu and Jia Zeng and Ping Luo and Jun Zhang and Andreas Geiger and Yu Qiao and Hongyang Li},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2024}
}

DriveLM

Introducing the First benchmark on Language Prompt for Driving.

Quick facts:

Task: given the language prompts as input, predict the trajectory in the scene
Origin dataset: nuScenes, CARLA (To be released)
Repo: https://github.com/OpenDriveLab/DriveLM, https://github.com/OpenDriveLab/ELM
Related work: DriveLM, ELM
Related challenge: Driving with Language AGC Challenge 2024

DriveData Survey

Abstract

With the continuous maturation and application of autonomous driving technology, a systematic examination of open-source autonomous driving datasets becomes instrumental in fostering the robust evolution of the industry ecosystem. In this survey, we provide a comprehensive analysis of more than 70 papers on the timeline, impact, challenges, and future trends in autonomous driving dataset.

Open-sourced Data Ecosystem in Autonomous Driving: the Present and Future

English Version

Chinese Version Accepted at SCIENTIA SINICA Informationis (中文版)

@article{li2024_driving_dataset_survey,
 title = {Open-sourced Data Ecosystem in Autonomous Driving: the Present and Future},
 author = {Hongyang Li and Yang Li and Huijie Wang and Jia Zeng and Huilin Xu and Pinlong Cai and Li Chen and Junchi Yan and Feng Xu and Lu Xiong and Jingdong Wang and Futang Zhu and Chunjing Xu and Tiancai Wang and Fei Xia and Beipeng Mu and Zhihui Peng and Dahua Lin and Yu Qiao},
 journal = {SCIENTIA SINICA Informationis},
 year = {2024},
 doi = {10.1360/SSI-2023-0313}
}

overview

Current autonomous driving datasets can broadly be categorized into two generations since the 2010s. We define the Impact (y-axis) of a dataset based on sensor configuration, input modality, task category, data scale, ecosystem, etc.

overview

Related Work Collection

We present comprehensive paper collections, leaderboards, and challenges.(Click to expand)

Challenges and Leaderboards

Title

Host

Year

Task

Entry

Autonomous Driving Challenge

OpenDriveLab

CVPR2023

Perception / OpenLane Topology

111

Perception / Online HD Map Construction

Perception / 3D Occupancy Prediction

Prediction & Planning / nuPlan Planning

Waymo Open Dataset Challenges

Waymo

CVPR2023

Perception / 2D Video Panoptic Segmentation

Perception / Pose Estimation

Prediction / Motion Prediction

Prediction / Sim Agents

CVPR2022

Prediction / Motion Prediction

128

Prediction / Occupancy and Flow Prediction

Perception / 3D Semantic Segmentation

Perception / 3D Camera-only Detection

CVPR2021

Prediction / Motion Prediction

115

Prediction / Interaction Prediction

Perception / Real-time 3D Detection

Perception / Real-time 2D Detection

Argoverse Challenges

Argoverse

CVPR2023

Prediction / Multi-agent Forecasting

Perception & Prediction / Unified Sensorbased Detection, Tracking, and Forecasting

Perception / LiDAR Scene Flow

Prediction / 3D Occupancy Forecasting

CVPR2022

Perception / 3D Object Detection

Prediction / Motion Forecasting

Perception / Stereo Depth Estimation

CVPR2021

Perception / Stereo Depth Estimation

368

Prediction / Motion Forecasting

Perception / Streaming 2D Detection

CARLA Autonomous Driving Challenge

CARLA Team, Intel

2023

Planning / CARLA AD Challenge 2.0

NeurIPS2022

Planning / CARLA AD Challenge 1.0

NeurIPS2021

Planning / CARLA AD Challenge 1.0

粤港澳大湾区（黄埔）国际算法算例大赛

琶洲实验室

2023

感知 / 跨场景单目深度估计

感知 / 路侧毫米波雷达标定和目标跟踪

2022

感知 / 路侧三维感知算法

感知 / 街景图像店面招牌文字识别

AI Driving Olympics

ETH Zurich, University of Montreal,Motional

NeurIP2021

Perception / nuScenes Panoptic

ICRA2021

Perception / nuScenes Detection

456

Perception / nuScenes Tracking

Prediction / nuScenes Prediction

Perception / nuScenes LiDAR Segmentation

计图 (Jittor)人工智能算法挑战赛

国家自然科学基金委信息科学部

2021

感知 / 交通标志检测

KITTI Vision Benchmark Suite

University of Tübingen

2012

Perception / Stereo, Flow, Scene Flow, Depth, Odometry, Object, Tracking, Road, Semantics

5,610

(back to top)

Perception Datasets

Dataset

Year

Diversity

Sensor

Annotation

Paper

Scenes

Hours

Region

Camera

Lidar

Other

KITTI

2012

Font-view

✗

GPS & IMU

2D BBox & 3D BBox

DriveAGI

Table of Contents

NEWS

GenAD: OpenDV Dataset

⭐ Generalized Predictive Model for Autonomous Driving (CVPR 2024, Highlight)

Paper | Video | Poster | Slides

Vista

🌏 A Generalizable Driving World Model with High Fidelity and Versatile Controllability (NeurIPS 2024)

DriveLM

DriveData Survey

Abstract

Related Work Collection

OpenScene

OpenLane-V2 Update