Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond 
[Zheng Zhu*](https://scholar.google.com/citations?user=NmwjI0AAAAAJ&hl=en),
[Xiaofeng Wang*](https://scholar.google.com.hk/citations?user=5IJ0Yg4AAAAJ&hl=zh-CN),
[Wangbo Zhao*](https://scholar.google.co.jp/citations?user=aocj89kAAAAJ&hl=es),
[Chen Min*](https://scholar.google.com/citations?user=pE9gTMQAAAAJ&hl=zh-CN),
[Bohan Li*](https://scholar.google.com/citations?user=V-YdQiAAAAAJ&hl=zh-CN),
[Nianchen Deng*](https://scholar.google.com/citations?user=AGPz8C4AAAAJ),
[Min Dou*](https://scholar.google.com.hk/citations?hl=zh-CN&user=w9fTWKQAAAAJ),
[Yuqi Wang*](https://scholar.google.com/citations?user=35UcX9sAAAAJ&hl=en),
[Botian Shi#](https://scholar.google.com.hk/citations?hl=zh-CN&user=K0PpvLkAAAAJ),
[Kai Wang########## denotes corresponding authors)
(Source:Sora, DriveDreamer, DriveDreamer-2, Drive-WM, UniSim, UniPi, RoboDreamer)
This is the official repository for the technical report:
Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond.
📌 Introduction
In our report, we present a holistic examination of recent advancements in world model research, encompassing profound philosophical perspectives and detailed discussions. Our analysis delves deeply into the literature surrounding world models for video generation, autonomous driving, and autonomous agents, uncovering their applications in media production, artistic expression, end-to-end driving, games, and robots. We assess the existing challenges and limitations of world models and delve into prospective avenues for future research, with the intention of steering and igniting further progress in world models.
Papers and Toolboxes for Video Generation World Models
| Methods | Task | Github |
|---|---|---|
| Open-Sora-Plan | T2V Generation |  |
| Open-Sora | T2V Generation |  |
| Sora | T2V Generation & Editing | - |
| IRC-GAN | T2V Generation | - |
| TGANs-C | T2V Generation | - |
| TFGANs | T2V Generation | - |
| StoryGAN | T2V Generation |  |
| TiVGAN | T2V Generation | - |
| GODIVA | T2V Generation |  |
| VideoGPT | C2V Generation |  |
| StoryDALL-E | C2V Generation | - |
| CogVideo | T2V Generation |  |
| Imagen Video | T2V Generation | - |
| MAGViT | C2V Generation |  |
| MAGViT-V2 | C2V Generation |  |
| VideoPoet | T2V Generation | - |
| SVD | T2V Generation |  |
| WorldDreamer | T2V Generation |  |
| Latte | T2V Generation |  |
| StreamingT2V | T2V Generation |  |
Papers and Toolboxes for Autonomous Driving World Models
| Methods | Task | Github |
|---|---|---|
| Iso-Dream | End-to-end Driving | - |
| MILE | End-to-end Driving |  |
| SEM2 | End-to-end Driving | - |
| TrafficBots | End-to-end Driving | - |
| Think2Drive | End-to-end Driving | - |
| GAIA-1 | Neural Driving Simulator (2D) | - |
| Tesla | Neural Driving Simulator | - |
| DriveDreamer | Neural Driving Simulator (2D) |  |
| ADriver-I | Neural Driving Simulator (2D) | - |
| DrivingDiffusion | Neural Driving Simulator (2D) | - |
| Panacea | Neural Driving Simulator (2D) |  |
| Drive-WM | Neural Driving Simulator (2D) & End-to-end Driving |  |
| WoVoGen | Neural Driving Simulator (2D) | - |
| DriveDreamer-2 | Neural Driving Simulator (2D) |  |
| GenAD | Neural Driving Simulator (2D) |  |
| SubjectDrive | Neural Driving Simulator (2D) | - |
| Copilot4D | Neural Driving Simulator (3D) | - |
| OccWorld | Neural Driving Simulator (3D) |  |
| MUVO | Neural Driving Simulator (3D) | - |
| LidarDM | Neural Driving Simulator (3D) | - |
| UniScene | Neural Driving Simulator (3D) |  |
| UniWorld | Neural Driving Simulator (3D) & 4D Pre-training | - |
| ViDAR | Neural Driving Simulator (3D) & 4D Pre-training |  |
| DriveWorld | Neural Driving Simulator (3D) & 4D Pre-training | - |
Papers and Toolboxes for Autonomous Agents World Models
| Methods | Task | Github |
|---|---|---|
| PlaNet | Robotics |  |
| World Models | Game Agent |  |
| RobotDreamPolicy | Robotics | - |
| Plan2Explore | Robotics |  |
| DreamerV1 | Robotics |  |
| SimPLe | Game Agent |  |
| Dreaming | Robotics | - |
| DreamerV2 | Game Agent |  |
| LEXA | Robotics |  |
| PathDreamer | Indoor Navigation |  |
| DreamerPro | Robotics |  |
| DreamingV2 | Robotics | - |
| TransDreamer | Game Agent & Robotics |  |
| IRIS | Game Agent |  |
| JEPA | Framework | - |
| Dr.G | Robotics |  |
| SWIM | Robotics | - |
| DreamerV3 | Game Agent & Robotics |  |
| HarmonyDream | Game Agent & Robotics | - |
| DayDreamer | Robotics |  |
| TWM | Game Agent |  |
| STORM | Game Agent |  |
| MC-JEPA | Optics Flow Prediction | - |
| A-JEPA | Audio Classification | - |
| I_JEPA | Image Semantics |  |
| SafeDreamer | Robotics |  |
| Genie | Generative Interactive Environment | - |
| V-JEPA | Video Prediction |  |
| ORV | Robotics |  |
| RoboDreamer | Robotics | - |
| UniSim | Generative Interactive Environment | - |
Contact
If you find our survey is useful in your research or applications, please consider giving us a star 🌟 and citing it by the following BibTeX entry.
@article{generalworldmodelsurvey,
title={Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond},
author={Zheng Zhu and Xiaofeng Wang and Wangbo Zhao and Chen Min and Nianchen Deng and Min Dou and Yuqi Wang and Botian Shi and Kai Wang and Chi Zhang and Yang You and Zhaoxiang Zhang and Dawei Zhao and Liang Xiao and Jian Zhao and Jiwen Lu and Guan Huang},
journal={arXiv preprint arXiv:2405.03520},
year={2024}
}