🔥🔥🔥 A curated list of papers on LLMs-based multimodal generation (image, video, 3D and audio).

LLMs Meet Multimodal Generation and Editing: A Survey

🤗 Introduction

This repository contains a curated list of LLMs meet multimodal generation. Modalities consist of visual (including image, video and 3D) and audio (including sound, speech and music).
We welcome any contributions and suggestions to our repository or the addition of your own work. Feel free to make a pull request or leave your comments!!

📋 Contents

🤗 Introduction
📋 Contents
💘 Tips
📍 Multimodal Generation
📍 Multimodal Editing
📍 Multimodal Agents
📍 Multimodal Understanding with LLMs
📍 Multimodal LLM Safety
📍 Related Surveys
- LLM
- Vision
👨‍💻 Team
😉 Citation
⭐️ Star History

💘 Tips

✅ Paper searching via catatogue: directly clicking the content of the catatogue to select the area of your research and browse related papers.
✅ Paper searching via author name: Free feel to search papers of a specific author via ctrl + F and then type the author name. The dropdown list of authors will automatically expand when searching.
✅ Paper searching via tag: You can also search the related papers via the following tags: customization, iteractive, human motion generation tokenizer. (More tags are ongoing)

📍 Multimodal Generation

Image Generation

🔅 LLM-based

I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models (12 Feb 2025)

Zhenxing Mi, Kuan-Chieh Wang, Guocheng Qian, et al. Zhenxing Mi, Kuan-Chieh Wang, Guocheng Qian, Hanrong Ye, Runtao Liu, Sergey Tulyakov, Kfir Aberman, Dan Xu

MetaMorph: Multimodal Understanding and Generation via Instruction Tuning (18 Dec 2024)

Shengbang Tong, David Fan, Jiachen Zhu, et al. Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, Zhuang Liu

X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Models (2 Dec 2024)

Zeyi Sun, Ziyang Chu, Pan Zhang, et al. Zeyi Sun, Ziyang Chu, Pan Zhang, Tong Wu, Xiaoyi Dong, Yuhang Zang, Yuanjun Xiong, Dahua Lin, Jiaqi Wang

Cosmos Tokenizer: A suite of image and video neural tokenizers (06 Nov 2024)

Fitsum Reda, Jinwei Gu, Xian Liu et al. Fitsum Reda, Jinwei Gu, Xian Liu, Songwei Ge, Ting-Chun Wang, Haoxiang Wang, Ming-Yu Liu

tokenizer

[ICLR 2025 Spotlight] Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance (29 Oct 2024)

Dongmin Park, Sebin Kim, Taehong Moon et al. Dongmin Park, Sebin Kim, Taehong Moon, Minkyu Kim, Kangwook Lee, Jaewoong Cho

ElasticTok: Adaptive Tokenization for Image and Video (10 Oct 2024)

Wilson Yan, Matei Zaharia, Volodymyr Mnih et al. Wilson Yan, Matei Zaharia, Volodymyr Mnih, Pieter Abbeel, Aleksandra Faust, Hao Liu

tokenizer

DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation (10 Oct 2024)

Jiatao Gu, Yuyang Wang, Yizhe Zhang et al. Jiatao Gu, Yuyang Wang, Yizhe Zhang, Qihang Zhang, Dinghuai Zhang, Navdeep Jaitly, Josh Susskind, Shuangfei Zhai

VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation (6 Sep 2024)

Yecheng Wu, Zhuoyang Zhang, Junyu Chen et al. Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, Song Han, Yao Lu

OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation (13 Jun 2024)

Junke Wang, Yi Jiang, Zehuan Yuan et al. Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zuxuan Wu, Yu-Gang Jiang

tokenizer

InstantUnify: Integrates Multimodal LLM into Diffusion Models (Aug 2024)

Qixun Wang, Xu Bai, Rui Wang et al. Qixun Wang, Xu Bai, Rui Wang, Haofan Wang

Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (22 Aug 2024)

Jinheng Xie, Weijia Mao, Zechen Bai, et al. Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, Mike Zheng Shou

Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptions (11 Jun 2024)

Renjie Pi, Jianshu Zhang, Jipeng Zhang et al. Renjie Pi, Jianshu Zhang, Jipeng Zhang, Rui Pan, Zhekai Chen, Tong Zhang

T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text (11 Jun 2024)

[ACL 2024] Aoxiong Yin, Haoyuan Li, Kai Shen et al. Aoxiong Yin, Haoyuan Li, Kai Shen, Siliang Tang, Yueting Zhuang

Open-World Human-Object Interaction Detection via Multi-modal Prompts (11 Jun 2024)

Jie Yang, Bingliang Li, Ailing Zeng et al. Jie Yang, Bingliang Li, Ailing Zeng, Lei Zhang, Ruimao Zhang

Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense? (11 Jun 2024)

Xingyu Fu, Muyu He, Yujie Lu et al. Xingyu Fu, Muyu He, Yujie Lu, William Yang Wang, Dan Roth

An Image is Worth 32 Tokens for Reconstruction and Generation (11 Jun 2024)

Qihang Yu, Mark Weber, Xueqing Deng et al. Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, Liang-Chieh Chen

TRINS: Towards Multimodal Language Models that Can Read (10 Jun 2024)

[CVPR 2024] Ruiyi Zhang, Yanzhe Zhang, Jian Chen et al. Ruiyi Zhang, Yanzhe Zhang, Jian Chen, Yufan Zhou, Jiuxiang Gu, Changyou Chen, Tong Sun

[LlamaGen] Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (10 Jun 2024)

Peize Sun, Yi Jiang, Shoufa Chen et al. Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, Zehuan Yuan

Chameleon: Mixed-Modal Early-Fusion Foundation Models (16 May 2024)

Chameleon Team

SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation (22 Apr 2024)

Yuying Ge, Sijie Zhao, Jinguo Zhu, et al. Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, Ying Shan

Graphic Design with Large Multimodal Model (22 Apr 2024)

Yutao Cheng, Zhao Zhang, Maoke Yang, et al. Yutao Cheng, Zhao Zhang, Maoke Yang, Hui Nie, Chunyuan Li, Xinglong Wu, and Jie Shao

PMG : Personalized Multimodal Generation with Large Language Models (7 Apr 2024)

Xiaoteng Shen, Rui Zhang, Xiaoyan Zhao, et al. Xiaoteng Shen, Rui Zhang, Xiaoyan Zhao, Jieming Zhu, Xi Xiao

MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control (19 Mar 2024)

Enshen Zhou, Yiran Qin, Zhenfei Yin, et al. Enshen Zhou, Yiran Qin, Zhenfei Yin, Yuzhou Huang, Ruimao Zhang, Lu Sheng, Yu Qiao, Jing Shao

ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (8 Mar 2024)

Xiwei Hu, Rui Wang, Yixiao Fang, et al. Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, Gang Yu

StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis (30 Jan 2024)

Zecheng Tang, Chenfei Wu, Zekai Zhang, et al. Zecheng Tang, Chenfei Wu, Zekai Zhang, Mingheng Ni, Shengming Yin, Yu Liu, Zhengyuan Yang, Lijuan Wang, Zicheng Liu, Juntao Li, Nan Duan

tokenizer

DiffusionGPT: LLM-Driven Text-to-Image Generation System (18 Jan 2024)

Jie Qin, Jie Wu, Weifeng Chen, et al. Jie Qin, Jie Wu, Weifeng Chen, Yuxi Ren, Huixia Li, Hefeng Wu, Xuefeng Xiao, Rui Wang, Shilei Wen

StarVector: Generating Scalable Vector Graphics Code from Images (17 Dec 2023)

Juan A. Rodriguez, Shubham Agarwal, Issam H. Laradji, et al. Juan A. Rodriguez, Shubham Agarwal, Issam H. Laradji, Pau Rodriguez, David Vazquez, Christopher Pal, Marco Pedersoli

VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation (14 Dec 2023)

Jinguo Zhu, Xiaohan Ding, Yixiao Ge, et al. Jinguo Zhu, Xiaohan Ding, Yixiao Ge, Yuying Ge, Sijie Zhao, Hengshuang Zhao, Xiaohua Wang, Ying Shan

StoryGPT-V: Large Language Models as Consistent Story Visualizers (13 Dec 2023)

Xiaoqian Shen, Mohamed Elhoseiny Xiaoqian Shen, Mohamed Elhoseiny

GENIXER: Empowering Multimodal Large Language Models as a Powerful Data Generator (11 Dec 2023)

Henry Hengyuan Zhao, Pan Zhou, Mike Zheng Shou Henry Hengyuan Zhao, Pan Zhou, Mike Zheng Shou

Customization Assistant for Text-to-image Generation (5 Dec 2023)

Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, et al. Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, Tong Sun

customization

ChatIllusion: Efficient-Aligning Interleaved Generation ability with Visual Instruction Model (29 Nov 2023)

Xiaowei Chi, Yijiang Liu, Zhengkai Jiang, et al. Xiaowei Chi, Yijiang Liu, Zhengkai Jiang, Rongyu Zhang, Ziyi Lin, Renrui Zhang, Peng Gao, Chaoyou Fu, Shanghang Zhang, Qifeng Liu, Yike Guo

DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedback (29 Nov 2023)

Jiao Sun, Deqing Fu, Yushi Hu, et al. Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Herrmann, Sjoerd van Steenkiste, Ranjay Krishna, Cyrus Rashtchian

COLE: A Hierarchical Generation Framework for Graphic Design (28 Nov 2023)

Peidong Jia, Chenxuan Li, Zeyu Liu, et al. Peidong Jia, Chenxuan Li, Zeyu Liu, Yichao Shen, Xingru Chen, Yuhui Yuan, Yinglin Zheng, Dong Chen, Ji Li, Xiaodong Xie, Shanghang Zhang, Baining Guo

TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering (28 Nov 2023)

Jingye Chen, Yupan Huang, Tengchao Lv, et al. Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei

LLMGA: Multimodal Large Language Model based Generation Assistant (27 Nov 2023)

Bin Xia, Shiyin Wang, Yingfan Tao, et al. Bin Xia, Shiyin Wang, Yingfan Tao, Yitong Wang, Jiaya Jia

Self-correcting LLM-controlled Diffusion Models (27 Nov 2023)

Tsung-Han Wu, Long Lian, Joseph E. Gonzalez, et al. Tsung-Han Wu, Long Lian, Joseph E. Gonzalez, Boyi Li, Trevor Darrell

[ParaDiffusion] Paragraph-to-Image Generation with Information-Enriched Diffusion Model (29 Nov 2023)

Weijia Wu, Zhuang Li, Yefei He, et al. Weijia Wu, Zhuang Li, Yefei He, Mike Zheng Shou, Chunhua Shen, Lele Cheng, Yan Li, Tingting Gao, Di Zhang, Zhongyuan Wang

Tokenize and Embed ALL for Multi-modal Large Language Models (8 Nov 2023)

Zhen Yang, Yingxue Zhang, Fandong Meng, et al. Zhen Yang, Yingxue Zhang, Fandong Meng, Jie Zhou

tokenizer

WordArt Designer: User-Driven Artistic Typography Synthesis using Large Language Models (20 Oct 2023)

Jun-Yan He, Zhi-Qi Cheng, Chenyang Li, et al. Jun-Yan He, Zhi-Qi Cheng, Chenyang Li, Jingdong Sun, Wangmeng Xiang, Xianhui Lin, Xiaoyang Kang, Zengke Jin, Yusen Hu, Bin Luo, Yifeng Geng, Xuansong Xie, Jingren Zhou

LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (16 Oct 2023)

[ICLR 2024] Hanan Gani, Shariq Farooq Bhat, Muzammal Naseer, et al. Hanan Gani, Shariq Farooq Bhat, Muzammal Naseer, Salman Khan, Peter Wonka

Making Multimodal Generation Easier: When Diffusion Models Meet LLMs (13 Oct 2023)

Xiangyu Zhao, Bo Liu, Qijiong Liu, et al. Xiangyu Zhao, Bo Liu, Qijiong Liu, Guangyuan Shi, Xiao-Ming Wu

Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generation (12 Oct 2023)

Zhengyuan Yang, Jianfeng Wang, Linjie Li, et al. Zhengyuan Yang, Jianfeng Wang, Linjie Li, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Lijuan Wang

OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluation (11 Oct 2023)

Jie An, Zhengyuan Yang, Linjie Li, et al. Jie An, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Lijuan Wang, Jiebo Luo

Mini-DALLE3: Interactive Text to Image by Prompting Large Language Models (11 Oct 2023)

Zeqiang Lai, Xizhou Zhu, Jifeng Dai, et al. Zeqiang Lai, Xizhou Zhu, Jifeng Dai, Yu Qiao, Wenhai Wang

[DALL-E 3] Improving Image Generation with Better Captions

James Betker, Gabriel Goh, Li Jing, et al. James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, Aditya Ramesh

MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens (3 Oct 2023)
Kaizhi Zheng, Xuehai He, Xin Eric Wang.
Making LLaMA SEE and Draw with SEED Tokenizer (2 Oct 2023)

Yuying Ge, Sijie Zhao, Ziyun Zeng, et al. Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, Ying Shan

tokenizer

InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (30 Sep 2023)

Yulu Gan, Sungwoo Park, Alexander Schubert, et al. Yulu Gan, Sungwoo Park, Alexander Schubert, Anthony Philippakis, Ahmed M. Alaa

InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition (26 Sep 2023)

Pan Zhang, Xiaoyi Dong, Bin Wang, et al. Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, Wenwei Zhang, Hang Yan, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, Jiaqi Wang

Text-to-Image Generation for Abstract Concepts (26 Sep 2023)

Jiayi Liao, Xu Chen, Qiang Fu, et al. Jiayi Liao, Xu Chen, Qiang Fu, Lun Du, Xiangnan He, Xiang Wang, Shi Han, Dongmei Zhang

DreamLLM: Synergistic Multimodal Comprehension and Creation (20 Sep 2023)

[ICLR 2024] Runpei Dong, Chunrui Han, Yuang Peng, et al. Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, Li Yi

SwitchGPT: Adapting Large Language Models for Non-Text Outputs (14 Sep 2023)
Wang, Xinyu, Bohan Zhuang, and Qi Wu.
NExT-GPT: Any-to-Any Multimodal LLM (11 Sep 2023)

Shengqiong Wu, Hao Fei, Leigang Qu, et al. Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, Tat-Seng Chua

LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation (9 Aug 2023)

Leigang Qu, Shengqiong Wu, Hao Fei, et al. ACM MM 2023 Leigang Qu, Shengqiong Wu, Hao Fei, Liqiang Nie, Tat-Seng Chua

Planting a SEED of Vision in Large Language Model (16 Jul 2023)

Yuying Ge, Yixiao Ge, Ziyun Zeng, et al. Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, Ying Shan

Generative Pretraining in Multimodality (11 Jul 2023)

Quan Sun, Qiying Yu, Yufeng Cui, et al. Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, Xinlong Wang

SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs (30 Jun 2023)

[NeurIPS 2023 Spotlight] Lijun Yu, Yong Cheng, Zhiruo Wang, et al. Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolfgang Macherey, Yanping Huang, David A. Ross, Irfan Essa, Yonatan Bisk, Ming-Hsuan Yang, Kevin Murphy, Alexander G. Hauptmann, Lu Jiang

Controllable Text-to-Image Generation with GPT-4 (29 May 2023)

Tianjun Zhang, Yi Zhang, Vibhav Vineet, et al. Tianjun Zhang, Yi Zhang, Vibhav Vineet, Neel Joshi, Xin Wang

Generating Images with Multimodal Language Models (26 May 2023)
[NeurIPS 2023] Koh, Jing Yu, Daniel Fried, and Ruslan Salakhutdinov.
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models (24 May 2023)

[NeurIPS 2023] Weixi Feng, Wanrong Zhu, Tsu-jui Fu, et al. Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, William Yang Wang

Visual Programming for Text-to-Image Generation and Evaluation (24 May 2023)
[NeurIPS 2023] Jaemin Cho, Abhay Zala, Mohit Bansal.
LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models (23 May 2023)

Long Lian, Boyi Li, Adam Yala, et al. Long Lian, Boyi Li, Adam Yala, Trevor Darrell

Interactive Data Synthesis for Systematic Vision Adaptation via LLMs-AIGCs Collaboration (22 May 2023)

Qifan Yu, Juncheng Li, Wentao Ye, et al. Qifan Yu, Juncheng Li, Wentao Ye, Siliang Tang, Yueting Zhuang

LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation (18 May 2023)

[NeurIPS 2023] Yujie Lu, Xianjun Yang, Xiujun Li, et al. Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, William Yang Wang

SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models (9 May 2023)

[ACM MM 2023] Shanshan Zhong, Zhongzhan Huang, Wushao Wen, et al. Shanshan Zhong, Zhongzhan Huang, Wushao Wen, Jinghui Qin, Liang Lin

Grounding Language Models to Images for Multimodal Inputs and Outputs (31 Jan 2023)
[ICML 2023] Koh, Jing Yu, Ruslan Salakhutdinov, and Daniel Fried.
[RPG-DiffusionMaster] Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (22 Jan 2024)

[ICML 2024] Ling Yang, Zhaochen Yu, Chenlin Meng, et al. Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, Bin Cui

RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models (20 Feb 2024)

Xinchen Zhang, Ling Yang, Yaqi Cai, et al. Xinchen Zhang, Ling Yang, Yaqi Cai, Zhaochen Yu, Kai-Ni Wang, Jiake Xie, Ye Tian, Minkai Xu, Yong Tang, Yujiu Yang, Bin Cui

Non-LLM-based (Clip/T5)

Edify Image: High-Quality Image Generation with Pixel Space Laplacian Diffusion Models (11 Nov 2024)

NVIDIA: Yuval Atzmon, Maciej Bala, Yogesh Balaji, et al. NVIDIA: Yuval Atzmon, Maciej Bala, Yogesh Balaji, Tiffany Cai, Yin Cui, Jiaojiao Fan, Yunhao Ge, Siddharth Gururani, Jacob Huffman, Ronald Isaac, Pooya Jannaty, Tero Karras, Grace Lam, J. P. Lewis, Aaron Licata, Yen-Chen Lin, Ming-Yu Liu, Qianli Ma, Arun Mallya, Ashlee Martino-Tarr, Doug Mendez, Seungjun Nah, Chris Pruett, Fitsum Reda, Jiaming Song, Ting-Chun Wang, Fangyin Wei, Xiaohui Zeng, Yu Zeng, Qinsheng Zhang

InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation (3 Apr 2024)

Haofan Wang, Matteo Spinelli, Qixun Wang, et al. Haofan Wang, Matteo Spinelli, Qixun Wang, Xu Bai, Zekui Qin, Anthony Chen

InstantID: Zero-shot Identity-Preserving Generation in Seconds (15 Jan 2024)

Qixun Wang, Xu Bai, Haofan Wang, et al. Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, Anthony Chen, Huaxia Li, Xu Tang, Yao Hu

PIXART-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis (30 Sep 2023)

[ICLR 2024] Junsong Chen, Jincheng Yu, Chongjian Ge, et al. Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, Zhenguo Li

TextDiffuser: Diffusion Models as Text Painters (18 May 2023)

[NeurIPS 2023] Jingye Chen, Yupan Huang, Tengchao Lv, et al. Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei

TiGAN: Text-Based Interactive Image Generation and Manipulation (Dec 2022)

[AAAI 2022] Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, et al. Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, Chris Tensmeyer, Tong Yu,Changyou Chen, Jinhui Xu, Tong Sun

Tags: iteractive

Multi-Concept Customization of Text-to-Image Diffusion (8 Dec 2022)

[CVPR 2023] Nupur Kumari, Bingliang Zhang, Richard Zhang, et al. Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, Jun-Yan Zhu

Tags: customization

DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (25 Aug 2022)

[CVPR 2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, et al. Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, Kfir Aberman

Tags: customization

An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2 Aug 2022)

Rinon Gal, Yuval Alaluf, Yuval Atzmon, et al. Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or

Tags: customization

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (23 May 2022)
[NeurIPS 2022]

Saharia, Chitwan Chan, William Saxena, Saurabh Li, Lala Whang, Jay Denton, Emily L Ghasemipour, Kamyar Gontijo Lopes, Raphael Karagol Ayan, Burcu Salimans, Tim others

High-Resolution Image Synthesis with Latent Diffusion Models (20 Dec 2021)
[CVPR 2022 (Oral)]

Rombach, Robin Blattmann, Andreas Lorenz, et al. Rombach, Robin Blattmann, Andreas Lorenz, Dominik Esser, Patrick Ommer, Bj{"o}rn

Datasets

MIMIC-IT: Multi-Modal In-Context Instruction Tuning (8 Jun 2023)

[NeurIPS 2023] Bo Li, Yuanhan Zhang, Liangyu Chen, et al. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, Ziwei Liu

[LAION-Glyph] GlyphControl: Glyph Conditional Control for Visual Text Generation (29 May 2023)

[NeurIPS 2023] Yukang Yang, Dongnan Gui, Yuhui Yuan, et al. Yukang Yang, Dongnan Gui, Yuhui Yuan, Weicong Liang, Haisong Ding, Han Hu, Kai Chen

[MARIO-10M] TextDiffuser: Diffusion Models as Text Painters (18 May 2023)

[NeurIPS 2023] Jingye Chen, Yupan Huang, Tengchao Lv, et al. Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei

DataComp: In search of the next generation of multimodal datasets (27 Apr 2023)

[NeurIPS 2023] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, et al. Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, Ludwig Schmidt

[LLava-instruct] Visual Instruction Tuning (17 Apr 2023)

[NeurIPS 2023] Haotian Liu, Chunyuan Li, Qingyang Wu, et al. Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee

Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text (14 Apr 2023)

[NeurIPS 2023] Wanrong Zhu, Jack Hessel, Anas Awadalla, et al. Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, Yejin Choi

Language Is Not All You Need: Aligning Perception with Language Models (27 Feb 2023)

[NeurIPS 2023] Shaohan Huang, Li Dong, Wenhui Wang, et al. Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei

COYO-700M: Image-Text Pair Dataset (31 Aug 2022)
LAION-5B: An open large-scale dataset for training next generation image-text models (16 Oct 2022)

[NeurIPS 2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, et al. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, Jenia Jitsev

LAION COCO: 600M SYNTHETIC CAPTIONS FROM LAION2B-EN (15 Sep 2022)

Christoph Schuhmann, Andreas Köpf , Theo Coombes, et al. Christoph Schuhmann, Andreas Köpf , Theo Coombes, Richard Vencu, Benjamin Trom , Romain Beaumont

[M3W] Flamingo: a Visual Language Model for Few-Shot Learning (29 Apr 2022)

[NeurIPS 2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, et al. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, Karen Simonyan

[LAION-FACE]General Facial Representation Learning in a Visual-Linguistic Manner (6 Dec 2021)

[NeurIPS 2021] Yinglin Zheng, Hao Yang, Ting Zhang, et al. Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, Fang Wen

[LAION-400M] Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs (3 Nov 2021)

[NeurIPS 2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, et al. Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, Aran Komatsuzaki

WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning (2 Mar 2021)

[SIGIR 2021] Krishna Srinivasan, Karthik Raman, Jiecao Chen, et al. Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, Marc Najork

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (17 Feb 2021)

[CVPR 2021] Soravit Changpinyo, Piyush Sharma, Nan Ding, et al. Soravit Changpinyo, Piyush Sharma, Nan Ding, Radu Soricut

[ALIGN] Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (11 Feb 2021)

[ICML 2021] Chao Jia, Yinfei Yang, Ye Xia, et al. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, Tom Duerig

[MS COCO] Microsoft COCO: Common Objects in Context (1 May 2014)

[ECCV 2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, et al. Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, Piotr Dollár

[Im2Text] Describing Images Using 1 Million Captioned Photographs (12 Dec 2011)
[NeurIPS 2011] Vicente Ordonez, Girish Kulkarni, Tamara Berg

Video Generation

🔅 LLM-based

Loong: Generating Minute-level Long Videos with Autoregressive Language Models (3 Oct 2024)

Yuqing Wang, Tianwei Xiong, Daquan Zhou, et al. Yuqing Wang, Tianwei Xiong, Daquan Zhou, Zhijie Lin, Yang Zhao, Bingyi Kang, Jiashi Feng, Xihui Liu

Compositional 3D-aware Video Generation with LLM Director (31 Aug 2024)

Hanxin Zhu, Tianyu He, Anni Tang, et al. Hanxin Zhu, Tianyu He, Anni Tang, Junliang Guo, Zhibo Chen, Jiang Bian

Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation (19 Aug 2024)

[SIGGRAPH Asia 2024] Yunxin Li, Haoyuan Shi, Baotian Hu, et al. Yunxin Li, Haoyuan Shi, Baotian Hu, Longyue Wang, Jiashun Zhu, Jinyi Xu, Zhen Zhao, Min Zhang

[BSQ-ViT] Image and Video Tokenization with Binary Spherical Quantization (11 Jun 2024)
[Tech Report]Yue Zhao, Yuanjun Xiong, Philipp Krähenbühl
tokenizer
DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation (11 Mar 2024)

Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, et al. Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, Xingang Wang

[Sora] Video generation models as world simulators (15 Feb 2024)

Tim Brooks, Bill Peebles, Connor Holmes, et al. Tim Brooks and Bill Peebles and Connor Holmes and Will DePue and Yufei Guo and Li Jing and David Schnurr and Joe Taylor and Troy Luhman and Eric Luhman and Clarence Ng and Ricky Wang and Aditya Ramesh

[LWM] World Model on Million-Length Video And Language With Blockwise RingAttention (13 Feb 2024)

Hao Liu, Wilson Yan, Matei Zaharia, et al. Hao Liu, Wilson Yan, Matei Zaharia, Pieter Abbeel

[LGVI] Towards Language-Driven Video Inpainting via Multimodal Large Language Models (18 Jan 2024)

Jianzong Wu, Xiangtai Li, Chenyang Si, et al. Jianzong Wu, Xiangtai Li, Chenyang Si, Shangchen Zhou, Jingkang Yang, Jiangning Zhang, Yining Li, Kai Chen, Yunhai Tong, Ziwei Liu, Chen Change Loy

Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization: Content-Consistent Multi-Scene Video Generation with LLM (2 Jan 2024)

Yang Jin, Zhicheng Sun, Kun Xu, et al. Yang Jin, Zhicheng Sun, Kun Xu, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, Kun Gai, Yadong Mu

tokenizer

VideoDrafter: Content-Consistent Multi-Scene Video Generation with LLM (2 Jan 2024)

Fuchen Long, Zhaofan Qiu, Ting Yao, et al. Fuchen Long, Zhaofan Qiu, Ting Yao, Tao Mei

[PRO-Motion] Plan, Posture and Go: Towards Open-World Text-to-Motion Generation (22 Dec 2023)

Jinpeng Liu, Wenxun Dai, Chunyu Wang, et al. Jinpeng Liu, Wenxun Dai, Chunyu Wang, Yiji Cheng, Yansong Tang, Xin Tong

VideoPoet: A Large Language Model for Zero-Shot Video Generation (21 Dec 2023)

Dan Kondratyuk, Lijun Yu, Xiuye Gu, et al. Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, Yong Cheng, Ming-Chang Chiu, Josh Dillon, Irfan Essa, Agrim Gupta, Meera Hahn, Anja Hauth, David Hendon, Alonso Martinez, David Minnen, David Ross, Grant Schindler, Mikhail Sirotenko, Kihyuk Sohn, Krishna Somandepalli, Huisheng Wang, Jimmy Yan, Ming-Hsuan Yang, Xuan Yang, Bryan Seybold, Lu Jiang

FlowZero: Zero-Shot Text-to-Video Synthesis with LLM-Driven Dynamic Scene Syntax (27 Nov 2023)

[arXiv 2023] Yu Lu, Linchao Zhu, Hehe Fan, et al. Yu Lu, Linchao Zhu, Hehe Fan, Yi Yang

InterControl: Generate Human Motion Interactions by Controlling Every Joint (27 Nov 2023)

Zhenzhi Wang, Jingbo Wang, Dahua Lin, et al. Zhenzhi Wang, Jingbo Wang, Dahua Lin, Bo Dai

Tags: human motion generation

MotionLLM: Multimodal Motion-Language Learning with Large Language Models (27 May 2024)

Qi Wu, Yubo Zhao, Yifan Wang, et al. Qi Wu, Yubo Zhao, Yifan Wang, Yu-Wing Tai, Chi-Keung Tang

Tags: general human motion generation

GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning (21 Nov 2023)

Jiaxi Lv, Yi Huang, Mingfu Yan, et al. Jiaxi Lv, Yi Huang, Mingfu Yan, Jiancheng Huang, Jianzhuang Liu, Yifan Liu, Yafei Wen, Xiaoxin Chen, Shifeng Chen

[LVD] LLM-grounded Video Diffusion Models (29 Sep 2023)

Long Lian, Baifeng Shi, Adam Yala, et al. Long Lian, Baifeng Shi, Adam Yala, Trevor Darrell, Boyi Li

VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning (26 Sep 2023)

[arXiv 2023] Han Lin, Abhay Zala, Jaemin Cho, et al. Han Lin, Abhay Zala, Jaemin Cho, Mohit Bansal

Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator (25 Sep 2023)

[NIPS 2023] Hanzhuo Huang, Yufan Feng, Cheng Shi, et al. Hanzhuo Huang, Yufan Feng, Cheng Shi, Lan Xu, Jingyi Yu, Sibei Yang

[Dysen-VDM] Empowering Dynamics-aware Text-to-Video Diffusion with Large Language Models (26 Aug 2023)

[CVPR 2024] Hao Fei, Shengqiong Wu, Wei Ji, et al. Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Tat-Seng Chua

[DirecT2V] Large Language Models are Frame-level Directors for Zero-shot Text-to-Video Generation (23 May 2023)

[arXiv 2023] Susung Hong, Junyoung Seo, Sunghwan Hong, et al. Susung Hong, Junyoung Seo, Sunghwan Hong, Heeseong Shin, Seungryong Kim

Text2Motion: From Natural Language Instructions to Feasible Plans (21 Mar 2023)

[Autonomous Robots 2023] Kevin Lin, Christopher Agia, Toki Migimatsu, et al. Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, Jeannette Bohg

Non-LLM-based

OSV: One Step is Enough for High-Quality Image to Video Generation (17 Sep 2024)

Xiaofeng Mao, Zhengkai Jiang, Fu-Yun Wang, et al. Xiaofeng Mao, Zhengkai Jiang, Fu-Yun Wang, Wenbing Zhu, Jiangning Zhang, Hao Chen, Mingmin Chi, Yabiao Wang

[PAB] Real-Time Video Generation with Pyramid Attention Broadcast (26 Jun 2024)

Xuanlei Zhao, Xiaolong Jin, Kai Wang, et al. Xuanlei Zhao, Xiaolong Jin, Kai Wang, Yang You

Video-Infinity: Distributed Long Video Generation (24 Jun 2024)

Zhenxiong Tan, Xingyi Yang, Songhua Liu, et al. Zhenxiong Tan, Xingyi Yang, Songhua Liu, Xinchao Wang

Pandora: Towards General World Model with Natural Language Actions and Video (12 Jun 2024)

Jiannan Xiang, Guangyi Liu, Yi Gu, et al. Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, Zhengzhong Liu, Eric P. Xing, Zhiting Hu

Text-Animator: Controllable Visual Text Video Generation (25 Jun 2024)

Lin Liu, Quande Liu, Shengju Qian, et al. Lin Liu, Quande Liu, Shengju Qian, Yuan Zhou, Wengang Zhou, Houqiang Li, Lingxi Xie, Qi Tian

MotionBooth: Motion-Aware Customized Text-to-Video Generation (25 Jun 2024)

Jianzong Wu, Xiangtai Li, Yanhong Zeng, et al. Jianzong Wu, Xiangtai Li, Yanhong Zeng, Jiangning Zhang, Qianyu Zhou, Yining Li, Yunhai Tong, Kai Chen

FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models (24 Jun 2024)

Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, et al. Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingqing He, Menghan Xia, Ziwei Liu

Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model (22 Jun 2024)

Min Zhao, Hongzhou Zhu, Chendong Xiang, et al. Min Zhao, Hongzhou Zhu, Chendong Xiang, Kaiwen Zheng, Chongxuan Li, Jun Zhu

Image Conductor: Precision Control for Interactive Video Synthesis (21 Jun 2024)

Yaowei Li, Xintao Wang, Zhaoyang Zhang, et al. Yaowei Li, Xintao Wang, Zhaoyang Zhang, Zhouxia Wang, Ziyang Yuan, Liangbin Xie, Yuexian Zou, Ying Shan

VIDEOSCORE: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation (21 Jun 2024)

Xuan He, Dongfu Jiang, Ge Zhang, et al. Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, Kai Wang, Quy Duc Do, Yuansheng Ni, Bohan Lyu, Yaswanth Narsupalli, Rongqi Fan, Zhiheng Lyu, Yuchen Lin, Wenhu Chen

Dreamitate: Real-World Visuomotor Policy Learning via Video Generation (24 Jun 2024)

Junbang Liang, Ruoshi Liu, Ege Ozguroglu, et al. Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sruthi Sudhakar, Achal Dave, Pavel Tokmakov, Shuran Song, Carl Vondrick

[MCM] Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation (11 Jun 2024)

Yuanhao Zhai, Kevin Lin, Zhengyuan Yang, et al. Yuanhao Zhai, Kevin Lin, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Chung-Ching Lin, David Doermann, Junsong Yuan, Lijuan Wang

Searching Priors Makes Text-to-Video Synthesis Better (5 Jun 2024)

Haoran Cheng, Liang Peng, Linxuan Xia, et al. Haoran Cheng, Liang Peng, Linxuan Xia, Yuepeng Hu, Hengjia Li, Qinglin Lu, Xiaofei He, Boxi Wu

ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video Generation (3 Jun 2024)

Shaoshu Yang, Yong Zhang, Xiaodong Cun, et al. Shaoshu Yang, Yong Zhang, Xiaodong Cun, Ying Shan, Ran He

EasyAnimate: A High-Performance Long Video Generation Method based on Transformer Architecture (30 May 2024)

Sijie Zhao, Yong Zhang, Xiaodong Cun, et al. Sijie Zhao, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Muyao Niu, Xiaoyu Li, Wenbo Hu, Ying Shan

[MOFT] Video Diffusion Models are Training-free Motion Interpreter and Controller (23 Mar 2024)

Zeqi Xiao, Yifan Zhou, Shuai Yang, et al. Zeqi Xiao, Yifan Zhou, Shuai Yang, Xingang Pan

StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (21 Mar 2024)

Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, et al. Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang, Shant Navasardyan, Humphrey Shi

Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis (22 Feb 2024)

Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, et al. Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, Sergey Tulyakov

VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models (17 Jan 2024)

Haoxin Chen, Yong Zhang, Xiaodong Cun, et al. Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, Ying Shan

Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (25 Nov 2023)

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, et al. Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, Robin Rombach

VideoCrafter1: Open Diffusion Models for High-Quality Video Generation (30 Oct 2023)

Haoxin Chen, Menghan Xia, Yingqing He, et al. Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, Ying Shan

DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors (18 Oct 2023)

Jinbo Xing, Menghan Xia, Yong Zhang, et al. Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsin Wong, Ying Shan

FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling (23 Oct 2023)

Haonan Qiu, Menghan Xia, Yong Zhang, et al. Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, Ziwei Liu

Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation (13 Jul 2023)

Yingqing He, Menghan Xia, Haoxin Chen, et al. Yingqing He, Menghan Xia, Haoxin Chen, Xiaodong Cun, Yuan Gong, Jinbo Xing, Yong Zhang, Xintao Wang, Chao Weng, Ying Shan, Qifeng Chen

Make-Your-Video: Customized Video Generation Using Textual and Structural Guidance (1 Jun 2023)

Jinbo Xing, Menghan Xia, Yuxin Liu, et al. Jinbo Xing, Menghan Xia, Yuxin Liu, Yuechen Zhang, Yong Zhang, Yingqing He, Hanyuan Liu, Haoxin Chen, Xiaodong Cun, Xintao Wang, Ying Shan, Tien-Tsin Wong

Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos (3 Apr 2023)

Yue Ma, Yingqing He, Xiaodong Cun, et al. Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Ying Shan, Xiu Li, Qifeng Chen

Real-time Controllable Denoising for Image and Video (29 Mar 2023)

[CVPR 2023] Zhaoyang Zhang, Yitong Jiang, Wenqi Shao, et al. Zhaoyang Zhang, Yitong Jiang, Wenqi Shao, Xiaogang Wang, Ping Luo, Kaimo Lin, Jinwei Gu

VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation (15 Mar 2023)

Zhengxiong Luo, Dayou Chen, Yingya Zhang, et al. Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, Tieniu Tan

Video VAE/Tokenizers

DLFR-VAE: Dynamic Latent Frame Rate VAE for Video Generation (17 Feb 2025)

Zhihang Yuan, Siyuan Wang, Rui Xie, et al. Zhihang Yuan, Siyuan Wang, Rui Xie, Hanling Zhang, Tongcheng Fang, Yuzhang Shang, Shengen Yan, Guohao Dai, Yu Wang

VideoVAE+: Large Motion Video Autoencoding with Cross-modal Video VAE (23 Dec 2024)

Yazhou Xing, Yang Fei, Yingqing He, et al. Yazhou Xing, Yang Fei, Yingqing He, Jingye Chen, Jiaxin Xie, Xiaowei Chi, Qifeng Chen

VidTwin: Video VAE with Decoupled Structure and Dynamics (23 Dec 2024)

Yuchi Wang, Junliang Guo, Xinyi Xie, et al. Yuchi Wang, Junliang Guo, Xinyi Xie, Tianyu He, Xu Sun, Jiang Bian

VidTok: A Versatile and Open-Source Video Tokenizer (17 Dec 2024)

Anni Tang, Tianyu He, Junliang Guo, et al. Anni Tang, Tianyu He, Junliang Guo, Xinle Cheng, Li Song, Jiang Bian

[CVPR 2025] WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model (26 Nov 2024)

Zongjian Li, Bin Lin, Yang Ye, et al. Zongjian Li, Bin Lin, Yang Ye, Liuhan Chen, Xinhua Cheng, Shenghai Yuan, Li Yuan

[CVPR 2025] [IV-VAE] Improved Video VAE for Latent Video Diffusion Model (10 Nov 2024)

Pingyu Wu, Kai Zhu, Yu Liu, et al. Pingyu Wu, Kai Zhu, Yu Liu, Liming Zhao, Wei Zhai, Yang Cao, Zheng-Jun Zha

[Tech Report] Cosmos Tokenizer: A suite of image and video neural tokenizers (Nov 6, 2024)

Fitsum Reda, Jinwei Gu, Xian Liu, et al. Fitsum Reda, Jinwei Gu, Xian Liu, Songwei Ge, Ting-Chun Wang, Haoxiang Wang, Ming-Yu Liu

[NeurIPS 2024] CV-VAE: A Compatible Video VAE for Latent Generative Video Models (30 May 2024)

Sijie Zhao, Yong Zhang, Xiaodong Cun, et al. Sijie Zhao, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Muyao Niu, Xiaoyu Li, Wenbo Hu, Ying Shan

[ICLR 2024] [MAGVIT-v2] Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (9 Oct 2023)

Lijun Yu, José Lezama, Nitesh B. Gundavarapu, et al. Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, Lu Jiang

Audio-Video

JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization (30 Mar 2025)

Kai Liu, Wei Li, Lai Chen, et al. Kai Liu, Wei Li, Lai Chen, Shengqiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Rongxin Jiang, Jiebo Luo, Hao Fei, Tat-Seng Chua

[LVAS-Agent] Long-Video Audio Synthesis with Multi-Agent Collaboration (13 Mar 2025)

Yehang Zhang, Xinli Xu, Xiaojie Xu, et al Yehang Zhang, Xinli Xu, Xiaojie Xu, Li Liu, Yingcong Chen

UniForm: A Unified Diffusion Transformer for Audio-Video Generation (6 Feb 2025)

Lei Zhao, Linfeng Feng, Dongxu Ge, et al Lei Zhao, Linfeng Feng, Dongxu Ge, Fangqiu Yi, Chi Zhang, Xiao-Lei Zhang, Xuelong Li

[![Paper](https://img.shields.io/badge/arXiv-b31b1b.svg)](https://arxiv.org/abs/2502.03897)
[![citation](https://img.shields.io/badge/citation-1-blue.svg?paper=7a4436209d8cf0d868b13000c8abff63d72daf0f)](https://www.semanticscholar.org/paper/UniForm%3A-A-Unified-Diffusion-Transformer-for-Zhao-Feng/7a4436209d8cf0d868b13000c8abff63d72daf0f)
[![Project_Page](https://img.shields.io/badge/Project_Page-00CED1)](https://uniform-t2av.github.io)

TIA2V: Video generation conditioned on triple modalities of text–image–audio (4 Jan 2025)

Minglu Zhao, Wenmin Wang, Rui Zhang, et al. Minglu Zhao, Wenmin Wang, Rui Zhang, Haomei Jia, Qi Chen

SAVGBench: Benchmarking Spatially Aligned Audio-Video Generation (18 Dec 2024)

Kazuki Shimada, Christian Simon, Takashi Shibuya, et al. Kazuki Shimada, Christian Simon, Takashi Shibuya, Shusuke Takahashi, Yuki Mitsufuji

[![Paper](https://img.shields.io/badge/arXiv-b31b1b.svg)](https://arxiv.org/abs/2412.13462)

AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation (19 Dec 2024)

Moayed Haji-Ali, Willi Menapace, Aliaksandr Siarohin, et al, Moayed Haji-Ali, Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Alper Canberk, Kwot Sin Lee, Vicente Ordonez, Sergey Tulyakov

[![Paper](https://img.shields.io/badge/arXiv-b31b1b.svg)](https://arxiv.org/abs/2412.15191)
[![Project_Page](https://img.shields.io/badge/Project_Page-00CED1)](https://snap-research.github.io/AVLink/)

SyncFlow: Temporally Aligned Joint Audio-Video Generation from Text (3 Dec 2024)

Haohe Liu, Gael Le Lan, Xinhao Mei, et al. Haohe Liu, Gael Le Lan, Xinhao Mei, Zhaoheng Ni, Anurag Kumar, Varun Nagaraja, Wenwu Wang, Mark D. Plumbley, Yangyang Shi, Vikas Chandra

[![Paper](https://img.shields.io/badge/arXiv-b31b1b.svg)](https://arxiv.org/abs/2412.15220)

A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generation (26 Sep 2024)

Masato Ishii, Akio Hayakawa, Takashi Shibuya Masato Ishii, Akio Hayakawa, Takashi Shibuya, Yuki Mitsufuji

[![Paper](https://img.shields.io/badge/arXiv-b31b1b.svg)](https://arxiv.org/abs/2409.17550)

AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation (11 Jun 2024)

Kai Wang, Shijian Deng, Jing Shi, et al. Kai Wang, Shijian Deng, Jing Shi, Dimitrios Hatzinakos, Yapeng Tian

[![Paper](https://img.shields.io/badge/arXiv-b31b1b.svg)](https://arxiv.org/abs/2406.07686)

Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation (28 May 2024)

Akio Hayakawa, Masato Ishii, Takashi Shibuya, et al. Akio Hayakawa, Masato Ishii, Takashi Shibuya, Yuki Mitsufuji

AudioScenic: Audio-Driven Video Scene Editing (25 Apr 2024)

Kaixin Shen, Ruijie Quan, Linchao Zhu, et al. Kaixin Shen, Ruijie Quan, Linchao Zhu, Jun Xiao, Yi Yang

A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation (22 May 2024)

Gwanghyun Kim, Alonso Martinez, Yu-Chuan Su, et al. Gwanghyun Kim, Alonso Martinez, Yu-Chuan Su, Brendan Jou, José Lezama, Agrim Gupta, Lijun Yu, Lu Jiang, Aren Jansen, Jacob Walker, Krishna Somandepalli

Semantically consistent Video-to-Audio Generation using Multimodal Language Large Model (25 Apr 2024)

Gehui Chen, Guan'an Wang, Xiaowen Huang, et al. Gehui Chen, Guan'an Wang, Xiaowen Huang, Jitao Sang

Awesome-LLMs-meet-Multimodal-Generation

🤗 Introduction

📋 Contents

💘 Tips

📍 Multimodal Generation

Image Generation

🔅 LLM-based

Non-LLM-based (Clip/T5)

Datasets

Video Generation

🔅 LLM-based

Non-LLM-based

Video VAE/Tokenizers

Audio-Video