STAR: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolution
[Rui Xie1*](https://github.com/CSRuiXie), 
[Yinhong Liu1*](https://github.com/yhliu04), 
[Penghao Zhou2](https://scholar.google.com/citations?hl=zh-CN&user=yWq1Fd4AAAAJ), 
[Chen Zhao1](https://scholar.google.com/citations?user=Uhp3JKgAAAAJ&hl=zh-CN&oi=sra), 
[Jun Zhou3](https://scholar.google.com/citations?hl=zh-CN&user=w03CHFwAAAAJ)
[Kai Zhang1](https://cszn.github.io/), 
[Zhenyu Zhang1](https://jessezhang92.github.io/), 
[Jian Yang1](https://scholar.google.com.hk/citations?user=6CIDtZQAAAAJ&hl=zh-CN), 
[Zhenheng Yang2](https://scholar.google.com/citations?hl=zh-CN&user=Ds5wwRoAAAAJ), 
[Ying Tai1†](https://tyshiwo.github.io/index.html)
1Nanjing University, 2ByteDance,  3Southwest University
[](https://nju-pcalab.github.io/projects/STAR)
[](https://arxiv.org/abs/2501.02976)
[](https://youtu.be/hx0zrql-SrU)
[](https://huggingface.co/spaces/SherryX/STAR)
[](https://colab.research.google.com/drive/1K8A1U_BNpAteRhhW9A8pAYs6LWjItQs_?usp=sharing)
🔆 Updates
-
2025.07.01 🔥 Training codes of I2VGen-XL version have been released.
-
2025.06.26 🌟 STAR is accepted by ICCV 2025!
-
2025.01.19 📖 The STAR demo is now available on Google Colab. Feel free to give it a try!
-
2025.01.09 📖 The online demo of STAR on Hugging Face is now live! Please note that due to the duration limitation of ZeroGPU, the running time may exceed the allocated GPU duration. If you'd like to try it, you can duplicate the demo and assign a paid GPU.
-
2025.01.07 🔥 The pretrained STAR model (I2VGen-XL and CogVideoX-5B versions) and inference code have been released.
📑 TODO
- [x] Training codes
- [x] Inference codes
- [x] Online demo
🔎 Method Overview
📷 Results Display
👀 More visual results can be found in our Project Page and Video Demo.
⚙️ Dependencies and Installation
VRAM requirement: Upscaling the provided toy example by 4x, with 72 frames, a width of 426, and a height of 240, requires around 39GB of VRAM using the default settings. If you encounter an OOM problem, you can set a smaller frame_length in inference_sr.sh. We recommend using a GPU with at least 24GB of VRAM to run this project.
## git clone this repository
git clone https://github.com/NJU-PCALab/STAR.git
cd STAR
## create an environment
conda create -n star python=3.10
conda activate star
pip install -r requirements.txt
sudo apt-get update && sudo apt-get install ffmpeg libsm6 libxext6 -y
🚀 Inference
Model Weight
| Base Model | Type | URL |
|---|---|---|
| I2VGen-XL | Light Degradation | :link: |
| I2VGen-XL | Heavy Degradation | :link: |
| CogVideoX-5B | Heavy Degradation | :link: |
1. I2VGen-XL-based
Step 1: Download the pretrained model STAR from HuggingFace.
We provide two versions for I2VGen-XL-based model, heavy_deg.pt for heavy degraded videos and light_deg.pt for light degraded videos (e.g., the low-resolution video downloaded from video websites).
You can put the weight into pretrained_weight/.
Step 2: Prepare testing data
You can put the testing videos in the input/video/.
As for the prompt, there are three options: 1. No prompt. 2. Automatically generate a prompt (e.g., using Pllava). 3. Manually write the prompt. You can put the txt file in the input/text/.
Step 3: Change the path
You need to change the paths in video_super_resolution/scripts/inference_sr.sh to your local corresponding paths, including video_folder_path, txt_file_path, model_path, and save_dir.
Step 4: Running inference command
bash video_super_resolution/scripts/inference_sr.sh
2. CogVideoX-based
Refer to these instructions for inference with the CogVideX-5B-based model.
Please note that the CogVideX-5B-based model supports only 720x480 input.
🌈 Training
Step 1: Download the pretrained VEnhancer.
Step 2: Prepare training data, structured as follows:
/dataset/
├── gt/ # Ground-truth high-quality videos
│ ├── video1.mp4
│ ├── video2.mp4
│ └── ...
├── lq/ # Low-quality input videos
│ ├── video1.mp4
│ ├── video2.mp4
│ └── ...
└── text/ # Text prompts corresponding to each video
├── video1.txt
├── video2.txt
└── ...
Follow this instruction to generate training data.
Step 3: Training for STAR (I2VGen-XL-based)
bash video_super_resolution/scripts/train_sr.sh
--pretrained_model_path: path to the pretrained VEnhancer.
--checkpointing_steps: save a model checkpoint every N training steps.
--num_frames: length of each training video.
❤️ Acknowledgments
This project is based on I2VGen-XL, VEnhancer, CogVideoX and OpenVid-1M. Thanks for their awesome works.
🎓Citations
If our project helps your research or work, please consider citing our paper:
@misc{xie2025starspatialtemporalaugmentationtexttovideo,
title={STAR: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolution},
author={Rui Xie and Yinhong Liu and Penghao Zhou and Chen Zhao and Jun Zhou and Kai Zhang and Zhenyu Zhang and Jian Yang and Zhenheng Yang and Ying Tai},
year={2025},
eprint={2501.02976},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2501.02976},
}
📧 Contact
If you have any inquiries, please don't hesitate to reach out via email at ruixie0097@gmail.com
📄 License
I2VGen-XL-based models are distributed under the terms of the MIT License.
CogVideoX-5B-based model is distributed under the terms of the CogVideoX License.