GPT4Scene-and-VLN-R1

VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning

[Zhangyang Qi](https://scholar.google.com/citations?user=kwVLpo8AAAAJ&hl=en/)1,2*, 
[Zhixiong Zhang](https://github.com/rookiexiong7)2, 
[Yizhou Yu](https://i.cs.hku.hk/~yzyu/publication/)1, 
[Jiaqi Wang](https://myownskyw7.github.io/)2✉,
[Hengshuang Zhao](https://hszhao.github.io/)1✉










    * Equation Contribution,
    ✉ Corresponding Authors,










1 The University of Hong Kong, 
2 Shanghai AI Laboratory,

[ICLR 2026] GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models

[Zhangyang Qi](https://scholar.google.com/citations?user=kwVLpo8AAAAJ&hl=en/)1,2*, 
[Zhixiong Zhang](https://github.com/rookiexiong7)2*, 
[Ye Fang](https://github.com/Aleafy)2, 
[Jiaqi Wang](https://myownskyw7.github.io/)2✉,
[Hengshuang Zhao](https://hszhao.github.io/)1✉










    * Equation Contribution,
    ✉ Corresponding Authors,










1 The University of Hong Kong, 
2 Shanghai AI Laboratory,

🔥 News

[2026/01/26] GPT4Scene is accepted to ICLR 2026!

[2025/07/06] We have released the training data, tokenizers and data generation code for VLN-R1.

[2025/06/20] We release the VLN-R1 paper in arxiv.

[2025/03/10] We release the training and validation dataset and training code.

[2025/01/21] We release the code validation datasetand model weights.

[2025/01/01] We release the GPT4Scene paper in arxiv. (The first paper in 2025! 🎇🎇🎇).

🌟 Note

This code currently focuses on GPT4Scene. We will merge the SFT section of VLN-R1 with GPT4Scene in the future. The following shows the open-source status of VLN-R1:

🔧 Installation

[!IMPORTANT] Installation is mandatory.

conda create --name gpt4scene python=3.10
conda activate gpt4scene

git clone https://github.com/Qi-Zhangyang/GPT4Scene.git
cd GPT4Scene

pip install -e ".[torch,metrics]"

Sometimes, the PyTorch downloaded this way may encounter errors. In such cases, you need to manually install Pytorch.

conda install pytorch==2.5.0 torchvision==0.20.0 torchaudio==2.5.0 pytorch-cuda=12.1 -c pytorch -c nvidia

pip install qwen_vl_utils flash-attn

🎡 Models and Weights

pip install --upgrade huggingface_hub
huggingface-cli login

🗂️ Dataset (ScanAlign)

You can download all trained model weights, dataset and annotations by

python download.py

The folder structure is as follows.

GPT4Scene
├── ckpts
│   ├── Qwen2-VL-7B-Instruct
├── data
│   ├── annotation
│   │   ├── images_2D
│   │   ├── images_3D
│   │   ├── sharegpt_data_chat_scene_32_images_3D_mark.json
├── evaluate
│   ├── annotation
│   │   ├── multi3dref_mask3d_val.json
│   │   ├── ...
│   │   └── sqa3d_val.json
│   ├── ...
│   └── utils
├── model_outputs
│   └── GPT4Scene-qwen2vl_full_sft_mark_32_3D_img512
├── ...
└── README.md

🚀 Inference

To inference, you can run the script

bash evaluate/infer.sh

It will automatically detect the number of GPUs in your current environment and perform chunked testing. Also you can use the slurm system to submit your evaluation task.

srun -p XXX --gres=gpu:4 --time=4-00:00:00 sh evaluate/infer.sh

🏗️ Training

You can start by getting the tokenizer - it only requires CPU resources.

bash gpt4scene_tokenizer_scripts/qwen2vl_7b_full_sft_mark_32_3D_img512.sh

Then you can run the training through 8 gpus.

bash gpt4scene_bash_scripts/qwen2vl_7b_full_sft_mark_32_3D_img512.sh

Also, you can run the torchrun code.

Please note that the initial run requires the tokenizer. It is recommended to disable the GPU initially and proceed with training after the tokenizer has completed.

⚖️ License

This repository is licensed under the Apache-2.0 License.

This repo benefits from LLaMA-Factory, Chat-Scene. Thanks for their wonderful works.

🔗 Citation

If this work is helpful, please kindly cite as:

@article{GPT4Scene,
  title={GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models},
  author={Zhangyang Qi and Zhixiong Zhang and Ye Fang and Jiaqi Wang and Hengshuang Zhao},
  journal={arXiv:2501.01428},
  year={2025}
}

@article{VLNR1,
  title={VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning},
  author={Zhangyang Qi and Zhixiong Zhang and Yizhou Yu and Jiaqi Wang and Hengshuang Zhao},
  journal={arXiv:2506.17221},
  year={2025}
}