← Open Source
tsinghua-fib-lab

World-Model

[ACM CSUR 2025] Understanding World or Predicting Future. A Comprehensive Survey of World Models.

ListsTool collectionsPaper collections
Open on GitHub
Momentum
+0stars in 24 hours0.0%
865
Stars
41
Forks
+4
This week
7
Contributors
Created 2025-06-08 · Updated 2026-10-15 · #13081 today
Top developers
zyk21Charles-DonneTHU-Yu-Shang
README

Awesome-World-Model

A curated list of awesome resources on World Models, based on the comprehensive survey "Understanding World or Predicting Future? A Comprehensive Survey of World Models".

News🔥

  • [2024/11/21] Initial release of our survey is available on arXiv.
  • [2025/06/13] Our survey paper "Understanding World or Predicting Future? A Comprehensive Survey of World Models" has been accepted by ACM Computing Surveys.
  • [2025/06/25] Second version of our survey is available on arXiv.
  • [2025/07/18] Initial release of the Awesome-World-Model GitHub repository.
  • [2025/11/18] Third version of our survey is available on arXiv.

Contact

If you have any suggestions or find our work helpful, feel free to contact us
Email: dingjt15@tsinghua.org.cn

If this list helps your research, please ⭐ and cite:

@article{ding2025understanding,
  title={Understanding World or Predicting Future? A Comprehensive Survey of World Models},
  author={Ding, Jingtao and Zhang, Yunke and Shang, Yu and Zhang, Yuheng and Zong, Zefang and Feng, Jie and Yuan, Yuan and Su, Hongyuan and Li, Nian and Sukiennik, Nicholas and others},
  journal={ACM Computing Surveys},
  volume={58},
  number={3},
  pages={1--38},
  year={2025},
  publisher={ACM New York, NY}
}

Table of Contents 🍃

Roadmap of world models in deep learning era

Model-based RL

Title Pub. & Date Code/Project URL
Recurrent world models facilitate policy evolution (RWM) NeurIPS 2018
Learning Latent Dynamics for Planning from Pixels (PlaNet) ICML 2019
Dream to control: Learning behaviors by latent imagination (Dreamer V1) ICLR 2020
Mastering atari with discrete world models (Dreamer V2) ICLR 2021
Temporal Difference Learning for Model Predictive Control (TD-MPC1) ICML 2023
Mastering Diverse Domains through World Models (Dreamer V3) 2023
TD-MPC2: Scalable, Robust World Models for Continuous Control (TD-MPC2) ICLR 2024
PWM: Policy Learning with Multi-Task World Models (PWM) ICLR 2025

Self-supervised learning

Title Pub.&Date Code/Project URL
A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27 (JEPA) 2024 —
DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning (DINO-WM) 2024
Revisiting Feature Prediction for Learning Visual Representations from Video (V-JEPA) 2024
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (V-JEPA2) 2025

LLM/MLLM

Title Pub.&Date Code/Project URL
Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning (LLM-DM) NeurIPS 2023
WorldGPT: Empowering LLM as Multimodal World Model (WorldGPT) ACM MM 2024
Text2World: Benchmarking Large Language Models for Symbolic World Model Generation (Text2World) ACL 2025

Video generation

Title Pub.&Date Code/Project URL
CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers (CogVideo) ICLR 2023
Structure and Content-Guided Video Synthesis with Diffusion Models (Gen‑1) ICCV 2023
UniSim: Learning Interactive Real-World Simulators (Unisim) ICLR 2024
Sora: Creating video from text (Sora) OpenAI 2024 —
World model on million-length video and language with ring-attention (LWM) ICLR 2025
Genie: Generative Interactive Environmentsn (Genie) ICML 2024
iVideoGPT: Interactive VideoGPTs are Scalable World Models (iVideoGPT) NeurIPS 2024
CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (CogVideoX) ICLR 2025
Wan: Open and Advanced Large-Scale Video Generative Models (Wan) 2025
Cosmos World Foundation Model Platform for Physical AI (Cosmos) 2025

Interactive 3D environment

Title Pub.&Date Code/Project URL
Interactive 3D Scene Generation from a Single Image (WonderWorld) CVPR 2025
Matrix-3D: Omnidirectional Explorable 3D World Generation (Matrix-3D) 2025
HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels (Hunyuan World) 2025

Application

Title Pub.&Date Code/Project URL
DayDreamer: World Models for Physical Robot Learning (DayDreamer) 2023
Generative Agents: Interactive Simulacra of Human Behavior (Generative Agents) UIST 2023
GAIA-1: A generative world model for autonomous driving (GAIA-1) 2023
OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving (OccWorld) ECCV 2024
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (GR1) ICLR 2024
DriveDreamer: Towards real-world-driven world models (DriveDreamer) ECCV 2024
Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (Drive-WM) CVPR 2024
Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2) (Think2Drive) ECCV 2024
Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (Drive-WM) CVPR 2024
GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving (GaussianWorld) CVPR 2025
Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (Drive-WM) 2025
World and Human Action Models towards Gameplay Ideation (WHAM) Nature 2025 —
Mineworld: a Real-time and Open-source Interactive World Model on Minecraft (MineWorld) 2025
GameFactory: Creating New Games with Generative Interactive Videos (Gamefactory) ICCV 2025
AgentSociety: Large-scale simulation of LLM-driven generative agents (AgentSociety) ACL 2025, COLM 2025
EnerVerse-AC: Envisioning Embodied Environments with Action Condition (EnerVerse) 2025
GR-3 Technical Report (GR3) 2025
Aether: Geometric-Aware Unified World Modeling (Aether) 2025
GWM: Towards Scalable Gaussian World Models for Robotic Manipulation (GWM) ICCV 2025
AirScape: An Aerial Generative World Model with Motion Controllability (AirScape) ACM MM 2025
RoboScape: Physics-informed Embodied World Model (RoboScape) 2025
DreamGen: Unlocking Generalization in Robot Learning through Video World Models (DreamGen) 2025
Matrix-Game: Interactive World Foundation Model (Matrix-Game) 2025
Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation (Genie Envisioner) 2025

3 Implicit Representation of the External World

3.1 World Model in Decision Making

Title Pub. & Date Code/Project URL
Deep reinforcement learning in a handful of trials using probabilistic dynamics models NeurIPS 2018
PWM: Policy Learning with Multi-Task World Models ICLR 2025
Recurrent world models facilitate policy evolution NeurIPS 2018
Dream to control: Learning behaviors by latent imagination ICLR 2020
Leveraging pre-trained large language models to construct and utilize world models for model-based task planning NeurIPS 2023
Mastering atari with discrete world models ICLR 2021
Mastering diverse control tasks through world models Nature 2024
TD-MPC2: Scalable, Robust World Models for Continuous Control ICLR 2024
When to trust your model: Model-based policy optimization NeurIPS 2019
Offline reinforcement learning as one big sequence modeling problem NeurIPS 2021
Model predictive control Springer —
Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees ICLR 2019
Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning ICRA 2018
A game theoretic framework for model based reinforcement learning ICML 2021
General agents need world models ICML 2025 —
Mastering memory tasks with world models ICLR 2024
A generalist dynamics model for control arXiv 2023 —
Exploring model-based planning with policy networks ICLR 2020
A0c: Alpha zero in continuous action space arXiv 2018
Probabilistic adaptation of text-to-video models ICLR 2024
RoboDreamer: Learning Compositional World Models for Robot Imagination ICML 2024
Discuss before moving: Visual language navigation via multi-expert discussions ICRA 2024
OVER-NAV: Elevating Iterative Vision-and-Language Navigation with Open-Vocabulary Detection and Structured Representation CVPR 2024
RILA: Reflective and Imaginative Language Agent for Zero-Shot Semantic Audio-Visual Navigation CVPR 2024
Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models arXiv 2025 —
Position: LLMs can't plan, but can help planning in LLM-modulo frameworks ICML 2024 —
Language models meet world models: Embodied experiences enhance language models NeurIPS 2023
Virtualhome: Simulating household activities via programs CVPR 2018
Learning to Model the World with Language ICML 2024
Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency ICML 2024
Alfworld: Aligning text and embodied environments for interactive learning ICLR 2021
Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents EMNLP 2024
Agent Planning with World Knowledge Model NeurIPS 2024
WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment NeurIPS 2024
Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation ICLR 2025

3.2 World Knowledge Learned by Models

Title Pub. & Date Code / Project URL
Does the chimpanzee have a theory of mind? Behav. & Brain Sci. 1978 —
GPT4GEO: How a Language Model Sees the World’s Geography NeurIPS 2023
LLMs achieve adult human performance on higher-order theory of mind tasks arXiv 2024 —
COKE: A cognitive knowledge graph for machine theory of mind ACL 2024
Think Twice: Perspective-Taking Improves LLM Theory-of-Mind ACL 2024
Language Models Represent Space and Time ICLR 2024
GeoLLM: Extracting Geospatial Knowledge from Large Language Models ICLR 2024
Large language models are geographically biased ICML 2024
Emergent Representations of Program Semantics in Language Models Trained on Programs ICML 2024
BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages NeurIPS 2024
SafeWorld: Geo-Diverse Safety Alignment NeurIPS 2024
EAI: Emotional Decision-Making of LLMs in Strategic Games and Ethical Dilemmas NeurIPS 2024
Testing theory of mind in large language models and humans Nature Human Behaviour 2024 Website
Automated construction of cognitive maps with visual predictive coding Nature Machine Intelligence 2024
Evaluating Large Language Models in Theory of Mind Tasks PNAS 2024 Website
Elements of World Knowledge (EWOK) Transactions of the ACL 2025 Website
The Geometry of Concepts: Sparse Autoencoder Feature Structure Entropy 2025
AgentMove: A large language model based agentic framework for zero-shot next location prediction NAACL 2025
CityGPT: Empowering Urban Spatial Cognition of Large Language Models KDD 2025
CityBench: Evaluating the Capabilities of Large Language Model as World Model KDD 2025
LocalGPT: Benchmarking and Advancing Large Language Models for Local Life Services KDD 2025
UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence ICCV 2025
Open-Set Living Need Prediction with Large Language Models ACL 2025 Findings
Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation ACL 2025 Findings Website
Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning EMNLP 2025 Findings
GPS as a Control Signal for Image Generation CVPR 2025
All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages CVPR 2025 Website
Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Models CVPR 2025
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces CVPR 2025 Website
A Survey of Large Language Model-Powered Spatial Intelligence Across Scales arXiv 2025 —
AI's Blind Spots: Geographic Knowledge and Diversity Deficit in Generated Urban Scenario arXiv 2025 —
Recognition through Reasoning: Reinforcing Image Geo-localization with Large Vision-Language Models arXiv 2025 —

4 Future Prediction of the Physical World

4.1 World Model as Video Generation

Title Pub. & Date Code / Project URL
Video generation models as world simulators OpenAI Blog 2024 —
Sora: Creating video from text OpenAI 2024 —
Is Sora a world simulator? A comprehensive survey on general world models and beyond arXiv 2024
Sora as an AGI world model? A complete survey on text-to-video generation arXiv 2024 —
How Far is Video Generation from World Model: A Physical Law Perspective ICML 2025
Do generative video models learn physical principles from watching videos? arXiv 2025
Genesis: A Generative and Universal Physics Engine for Robotics and Beyond ICML 2024
PhysGen: Rigid-body physics-grounded image-to-video generation ECCV 2024
NUWA-XL: Diffusion over Diffusion for Extremely Long Video Generation ACL 2023 Website
OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving ECCV 2024
OccSora: 4D Occupancy Generation Models as World Simulators ICLR 2025
World model on million-length video and language with ring-attention ICLR 2025
GAIA-1: A generative world model for autonomous driving arXiv 2023 Website
DriveDreamer: Towards real-world-driven world models ECCV 2024
DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation AAAI 2025
Driving into the Future: Multiview Visual Forecasting and Planning with World Model CVPR 2024
Vista: A Generalizable Driving World Model with High Fidelity NeurIPS 2024
WorldDreamer: Towards general world models for video generation arXiv 2024
WorldGPT: a Sora-inspired video AI agent arXiv 2024 —

4.2 World Model as Embodied Environment

Title Pub. & Date Code / Project URL
Holodeck: Language guided generation of 3d embodied ai environments CVPR 2024
GRUtopia: Dream General Robots in a City at Scale arXiv 2024
Anyhome: Open-vocabulary generation of structured and textured 3d homes ECCV 2024
LEGENT: Open Platform for Embodied Agents arXiv 2024
UrbanWorld: An Urban World Model for 3D City Generation arXiv 2024
MetaUrban: An Embodied AI Simulation Platform for Urban Micromobility ICLR 2025
Minedojo: Building open-ended embodied agents with internet-scale knowledge NeurIPS 2022
UniSim: Learning Interactive Real-World Simulators ICLR 2024
EmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City Environment arXiv 2024
Empowering World Models with Reflection for Embodied Video Prediction ICML 2025 —
Streetscapes: Large-scale consistent street view generation using autoregressive video diffusion SIGGRAPH 2024 —
AVID: Adapting Video Diffusion Models to World Models arXiv 2024
Pandora: Towards General World Model with Natural Language Actions and Video States arXiv 2024
RoboScape: Physics-informed Embodied World Model arXiv 2025
TesserAct: Learning 4D Embodied World Models arXiv 2025

5 Applications of World Models

5.1 Game Intelligence

Title Pub. & Date Code / Project URL
World and Human Action Models towards Gameplay Ideation Nature 2025 —
GameFactory: Creating New Games with Generative Interactive Videos ICCV 2025
Unbounded: A Generative Infinite Game of Character Life Simulation CVPR 2025 Website
GameGen-𝕏: Interactive Open-world Game Video Generation ICLR 2025
Diffusion Models Are Real-Time Game Engines ICLR 2025 Website
Exploration-Driven Generative Interactive Environments ICLR 2025
Matrix-Game: Interactive World Foundation Model arXiv 2025
Mineworld: a Real-time and Open-source Interactive World Model on Minecraft arXiv 2025
Model as a Game: On Numerical and Spatial Consistency for Generative Games arXiv 2025 —

5.2 Embodied Intelligence

Title Pub. & Date Code / Project URL
OpenEQA: Embodied Question Answering in the Era of Foundation Models CVPR 2024
iVideoGPT: Interactive VideoGPTs are Scalable World Models NeurIPS 2024
IRASim: A Fine-Grained World Model for Robot Manipulation ICCV 2025
RoboScape: Physics-informed Embodied World Model arXiv 2025
TesserAct: Learning 4D Embodied World Models arXiv 2025
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning arXiv 2025
Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations ICML 2025
DreamGen: Unlocking Generalization in Robot Learning through Video World Models arXiv 2025
EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation arXiv 2025 Website
EnerVerse-AC: Envisioning Embodied Environments with Action Condition arXiv 2025
Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation arXiv 2025
Vidar: Embodied Video Diffusion Model for Generalist Bimanual Manipulation arXiv 2025 —
WorldVLA: Towards Autoregressive Action World Model arXiv 2025
ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Model arXiv 2025
ORV: 4D Occupancy-centric Robot Video Generation arXiv 2025
GWM: Towards Scalable Gaussian World Models for Robotic Manipulation ICCV 2025 Website
WorldEval: World Model as Real-World Robot Policies Evaluator arXiv 2025

5.3 Urban Intelligence

Autonomous Driving

Title Pub. & Date Code / Project URL
Video generation models as world simulators OpenAI Research (2024) Website
GPT-4 technical report arXiv 2023 —
Visual Instruction Tuning NeurIPS 2023
World models for autonomous driving: An initial survey IEEE T-IV 2024 —
Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research arXiv 2023
Planning-oriented autonomous driving CVPR 2023
A survey on trajectory-prediction methods for autonomous driving IEEE T-IV 2022 —
BEVFormer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers ECCV 2022
Transfusion: Robust lidar-camera fusion for 3D object detection with transformers CVPR 2022
YOLOP: You only look once for panoptic driving perception MIR 2022
Wayformer: Motion forecasting via simple & efficient attention networks ICRA 2023 —
Motion Transformer with Global Intention Localization and Local Movement Refinement NeurIPS 2022
Query-Centric Trajectory Prediction CVPR 2023
HPTR: Real-time motion prediction via heterogeneous polyline transformer with relative pose encoding NeurIPS 2023
MotionDiffuser: Controllable multi-agent motion prediction using diffusion CVPR 2023 —
Tokenize the world into object-level knowledge to address long-tail events in autonomous driving arXiv 2024 —
GAIA-1: A generative world model for autonomous driving arXiv 2023 Website
DriveDreamer: Towards real-world-driven world models for autonomous driving ECCV 2024
Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving CVPR 2024
OccWorld: Learning a 3D occupancy world model for autonomous driving ECCV 2024
OccSora: 4D occupancy generation models as world simulators for autonomous driving arXiv 2024
Vista: A generalizable driving world model with high fidelity and versatile controllability NeurIPS 2024
Copilot4D: Learning unsupervised world models for autonomous driving via discrete diffusion ICLR 2024 —
MUVO: A multimodal generative world model for autonomous driving with geometric representations IEEE T-IV 2025
UniWorld: Autonomous driving pre-training via world models arXiv 2023
MetaUrban: A simulation platform for embodied AI in urban spaces ICLR 2025
UrbanWorld: An urban world model for 3D city generation arXiv 2024
Streetscapes: Large-scale consistent street view generation using autoregressive video diffusion SIGGRAPH 2024 Website

Autonomous Logistics & Urban Analytics

Title Pub. & Date Code / Project URL
Navigation World Models CVPR 2025
Towards Autonomous Micromobility through Scalable Urban Simulation CVPR 2025
Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigation CVPR 2025
CityWalker: Learning Embodied Urban Navigation from Web-Scale Videos CVPR 2025
AirScape: An Aerial Generative World Model with Motion Controllability (AirScape) ACM MM 2025
CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memory ACL 2025
CityEQA: A Hierarchical LLM Agent on Embodied Question Answering Benchmark in City Space EMNLP 2025
UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces ACL 2025
GeoLLM: Extracting Geospatial Knowledge from Large Language Models ICLR 2024
CityGPT: Empowering Urban Spatial Cognition of Large Language Models KDD 2025
UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence ICCV 2025
GPS as a Control Signal for Image Generation CVPR 2025
AI's Blind Spots: Geographic Knowledge and Diversity Deficit in Generated Urban Scenario arXiv 2025 —
AgentMove: A Large Language Model based Agentic Framework for Zero-shot Next Location Prediction NAACL 2025
CAMS: A CityGPT-Powered Agentic Framework for Urban Human Mobility Simulation arXiv 2025
Open-Set Living Need Prediction with Large Language Models ACL 2025

5.4 Societal Intelligence

Title Pub. & Date Code / Project URL
AgentSociety: Large-scale simulation of LLM-driven generative agents ACL 2025, COLM 2025
GenSim: A General Social Simulation Platform with Large Language Model based Agents NAACL 2025
Simulating Human-like Daily Activities with Desire-driven Autonomy ICLR 2025
EconAgent: Large language model-empowered agents for simulating macroeconomic activities ACL 2024
Agent-Pro: Learning to evolve via policy-level reflection and optimization ACL 2024
Exploring collaboration mechanisms for LLM agents: A social psychology view ACL 2024
Cooperate or Collapse: Emergence of sustainability behaviors in a society of LLM agents NeurIPS 2024
SocioDojo: Building Lifelong Analytical Agents with Real-world Text and Time Series ICLR 2024
SRAP-Agent: Simulating and optimizing scarce resource allocation policy with LLM-based agent EMNLP 2024
Generative agents: Interactive simulacra of human behavior UIST 2023
SocioVerse: A World Model for Social Simulation Powered by LLM Agents and A Pool of 10 Million Real-World Users arXiv 2025
YuLan-OneSim: Towards the Next Generation of Social Simulator with Large Language Models arXiv 2025
OASIS: Open Agent Social Interaction Simulations with One Million Agents arXiv 2024
Project Sid: Many-agent simulations toward AI civilization arXiv 2024
Network Formation and Dynamics Among Multi-LLMs arXiv 2024
S3: Social-network Simulation System with Large Language Model-Empowered Agents arXiv 2023 —
Exploring large language models for communication games: An empirical study on werewolf arXiv 2023
Nearby in the ranking
Similar items