Yifei Dong^1,*, Fengyi Wu^1,*, Qi He¹, Lingdong Kong², Heng Li¹, Minghan Li¹, Zebang Cheng¹, Yuxuan Zhou¹, Jingdong Sun³, Qi Dai⁴, Alexander G. Hauptmann³, Zhi-Qi Cheng^1,†
1UW, ²NUS, ³CMU, ⁴Microsoft Research

• [2026-09] 🏆 We are organizing the HA-VLN track of the RoboWorld Challenge 2026, affiliated with the RoboPAD Workshop at NeurIPS 2026. Join us in advancing human-aware navigation, participants from all backgrounds are welcome!
• [2026-06] 🎉 HA-VLN 2.0 has been accepted to IROS 2026!
• [2025-08] We have substantially upgraded the repository and improved usability, and organized a small-scale internal competition to support testing and community feedback.
• [2025-03] 🚀 We release HA-VLN 2.0, unifying discrete and continuous human-aware navigation with HAPS 2.0, dynamic multi-human interactions, and social-awareness evaluation across 16,844 instructions, check our Technical Report!
• [2024-09] 🎉 HA-VLN 1.0 has been accepted to the NeurIPS 2024 Datasets and Benchmarks Track!

Navigation Demo 1	Navigation Demo 2
Navigation Instruction: Start by moving forward in the lounge area, where an individual is engaged in a phone conversation while pacing back and forth. Navigate carefully to avoid crossing their path... (Click to expand) As you proceed, you will pass by a television mounted on the wall. Continue your movement, observing people relaxing and watching the TV, some seated comfortably on sofas. Further along, notice a group of friends raising their glasses in a toast, enjoying cocktails together. Maintain a steady course, ensuring you do not disrupt their gathering. Finally, reach the end of your path where a potted plant is situated next to a door. Stop at this location, positioning yourself near the plant and door without obstructing access.	Navigation Instruction: Exit the room and make a left turn. Proceed down the hallway where an individual is ironing clothes, carefully smoothing out wrinkles on garments. Continue walking and make another left turn... (Click to expand) Enter the next room, which is a bedroom. Inside, someone is comfortably seated in bed, engrossed in reading a book. Move past the bed, ensuring not to disturb the reader. Turn left again to enter the bathroom. Once inside, position yourself near the sink and wait there, observing the surroundings without interfering with any activities.

If you find this repository or our paper useful, please consider starring this repository and citing our paper. You are also welcome to explore our recent works on world modeling for navigation, including LCVN (NeurIPS 2026 Oral), UniWM (ECCV 2026), and GOViG (ACL 2026).

@inproceedings{dong2026havln,
  title={{HA-VLN 2.0: An Open Benchmark and Leaderboard for Human-Aware Navigation in Discrete and Continuous Environments with Dynamic Multi-Human Interactions}},
  author={Dong, Yifei and Wu, Fengyi and He, Qi and Kong, Lingdong and Li, Heng and Li, Minghan and Cheng, Zebang and Zhou, Yuxuan and Sun, Jingdong and Dai, Qi and Hauptmann, Alexander G. and Cheng, Zhi-Qi},
  booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year={2026},
  url={https://arxiv.org/abs/2503.14229}
}

We present Human-Aware Vision-and-Language Navigation (HA-VLN), expanding VLN to include both discrete (HA-VLN-DE) and continuous (HA-VLN-CE) environments with social behaviors. The HA-VLN Simulator enables real-time rendering of human activities and provides unified APIs for navigation development. It introduces the Human Activity and Pose Simulation (HAPS 2.0 Dataset) with detailed 3D human motion models and the HA Room-to-Room (HA-R2R) Dataset with complex navigation instructions that include human activities. We propose an HA-VLN Vision-and-Language model (HA-VLN-VL) and a Cross-Model Attention model (HA-VLN-CMA) to address visual-language understanding and dynamic decision-making challenges.

• HA-VLN
  • Table of Contents
  • 🚀 Quick Start
  • 📥 Download Dataset
  • 🔄 Dataset Organization
  • 🖥️ Real-time Human Rendering
  • 🌆 Human-Scene Fusion
  • 🌆 Navigate within a Scene Yourself
  • 📊 Training
  • 📈 Visualization
  • 📂 Dataset Details

In this section, you will download the necessary datasets, set up the Docker environment, and reproduce our proposed HA-VLN-CMA baseline model.

git clone https://github.com/UWMILab/HA-VLN.git
cd HA-VLN

All scene meshes, human activities, and baseline checkpoints reside in Data/:

# 1. Obtain download_mp.py after Matterport3D access approval:
# https://niessner.github.io/Matterport/
python3 /path/to/download_mp.py -o Data/scene_datasets --task_data habitat
# After task-data download, press Ctrl-C at the main-dataset prompt.
unzip Data/scene_datasets/v1/tasks/mp3d_habitat.zip -d Data/scene_datasets

# 2. 1-Click download validation episodes, HAPS 2.0, annotations, and CMA weights
python scripts/download_hf.py --destination Data --target all

# 3. Set up the released CMA baseline checkpoint
mkdir -p agent/VLN-CE/data/checkpoints/cma_pm_da_aug_tune
cp Data/checkpoints/HA-VLN-CMA/ckpt.39.pth \
  agent/VLN-CE/data/checkpoints/cma_pm_da_aug_tune/CMA_PM_DA_Aug.pth

With Docker and NVIDIA Container Toolkit installed, run evaluation on val_unseen:

IMAGE=ghcr.io/jostarxiong/havln-challenge-2026@sha256:78a62cd176d2fd7d0e2825f4cb5be2488ebc5f1a354649b7b4f536a98f1054f4
docker pull "$IMAGE"

DATA_DIR="$(cd Data && pwd -P)"
docker run --gpus all -it --rm \
  --shm-size 16g \
  --mount type=bind,source="$(pwd)",target=/workspace/HA-VLN \
  --mount type=bind,source="$DATA_DIR",target=/workspace/HA-VLN/Data \
  --mount type=bind,source="$DATA_DIR",target=/data/havln2 \
  --workdir /workspace/HA-VLN \
  "$IMAGE" bash -lc 'bash scripts/setup_docker_cma.sh && cd agent &&
    python run.py --exp-config config/cma_pm_da_aug_tune.yaml --run-type eval \
      MODEL.DEPTH_ENCODER.ddppo_checkpoint NONE VIDEO_OPTION "[]"'

(Tip: To evaluate val_seen, append EVAL.SPLIT val_seen to the Python command. To launch an interactive container, replace the final bash -lc ... with bash).

The trainer reports component metrics; use the participant toolkit for official action replay and Score.

HA_VLN_ROOT="$(pwd)"
conda create -n havlnce python=3.8 pip=24.0 -c conda-forge -y
conda activate havlnce
conda install -c aihabitat -c conda-forge \
  "habitat-sim=0.1.7=*headless*" "numpy=1.23.5" \
  python-lmdb libxcrypt libopengl libglx -y

python -m pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 \
  --index-url https://download.pytorch.org/whl/cu118

git clone --branch v0.1.7 --depth 1 \
  https://github.com/facebookresearch/habitat-lab.git habitat-lab
python -m pip install -c requirements-py38.txt \
  -r habitat-lab/requirements.txt setuptools pytest-runner \
  tensorboard moviepy webdataset ifcfg msgpack_numpy
python -m pip install --no-deps --no-build-isolation -e habitat-lab

# Agent dependencies
python -m pip install -r requirements-py38.txt

export LD_LIBRARY_PATH="$CONDA_PREFIX/lib:${LD_LIBRARY_PATH:-}"
export DISPLAY=""
export EGL_DEVICE_ID=0

sudo apt-get update
sudo apt-get install -y --no-install-recommends \
  cmake build-essential libjpeg-dev libglm-dev libgl1 \
  libegl1-mesa-dev mesa-utils xorg-dev freeglut3-dev
git clone --branch v0.1.7 --recursive \
  https://github.com/facebookresearch/habitat-sim.git habitat-sim
cd habitat-sim
python -m pip install -r requirements.txt -c "$HA_VLN_ROOT/requirements-py38.txt"
python setup.py install --headless
cd "$HA_VLN_ROOT"

source /opt/conda/etc/profile.d/conda.sh
conda activate havlnce
HA_VLN_ROOT=/workspace/HA-VLN
apt-get update
apt-get install -y git

cd "$HA_VLN_ROOT"
python -m pip install -r requirements-dino-py38.txt
conda install -c nvidia/label/cuda-11.8.0 -c conda-forge \
  cuda-toolkit gcc_linux-64=11 gxx_linux-64=11 sysroot_linux-64=2.17 -y
export CUDA_HOME="$CONDA_PREFIX"
export CC="$CONDA_PREFIX/bin/x86_64-conda-linux-gnu-gcc"
export CXX="$CONDA_PREFIX/bin/x86_64-conda-linux-gnu-g++"

git clone https://github.com/IDEA-Research/GroundingDINO.git HASimulator/GroundingDINO
git -C HASimulator/GroundingDINO checkout df5b48a3efbaa64288d8d0ad09b748ac86f22671
MAX_JOBS=2 python -m pip install --no-deps --no-build-isolation \
  -e HASimulator/GroundingDINO

mkdir -p HASimulator/GroundingDINO/weights
curl -fL --retry 3 \
  https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth \
  -o HASimulator/GroundingDINO/weights/groundingdino_swint_ogc.pth
python -m pip check

HA_VLN_ROOT="$(pwd)"
conda create -n havlnce python=3.7
conda activate havlnce
conda install -c aihabitat -c conda-forge habitat-sim=0.1.7 headless
git clone --branch v0.1.7 https://github.com/facebookresearch/habitat-lab.git
cd habitat-lab
pip install -r requirements.txt
pip install -r habitat_baselines/rl/requirements.txt
python setup.py develop --all
cd "$HA_VLN_ROOT"

# Agent packages (Python 3.7)
pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt

cd "$HA_VLN_ROOT"
# These pinned DINO dependencies also support Python 3.7 (supervision==0.11.1).
python -m pip install -r requirements-dino-py38.txt
git clone https://github.com/IDEA-Research/GroundingDINO.git HASimulator/GroundingDINO
git -C HASimulator/GroundingDINO checkout df5b48a3efbaa64288d8d0ad09b748ac86f22671
# Requires a system CUDA 11.1 toolkit and a compatible host compiler.
export CUDA_HOME=/usr/local/cuda-11.1
export PATH="$CUDA_HOME/bin:$PATH"
MAX_JOBS=2 python -m pip install --no-deps --no-build-isolation \
  -e HASimulator/GroundingDINO

mkdir -p HASimulator/GroundingDINO/weights
curl -fL --retry 3 \
  https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth \
  -o HASimulator/GroundingDINO/weights/groundingdino_swint_ogc.pth

Request access on the Matterport3D dataset page: sign its Terms of Use and send it to matterport3d@googlegroups.com. Use the helper supplied after approval; it is not distributed here. For the locally verified Python 3 helper:

python3 /path/to/download_mp.py -o Data/scene_datasets --task_data habitat
# After task-data download, press Ctrl-C at the main-dataset prompt.
unzip Data/scene_datasets/v1/tasks/mp3d_habitat.zip -d Data/scene_datasets

The final scene meshes should reside at Data/scene_datasets/mp3d/<scan>/<scan>.glb. Raw --type matterport_mesh ... downloads do not provide this Habitat layout. If your supplied helper requires Python 2, use that interpreter.

We provide two download options:

For the validation baseline, use scripts/download_hf.py as shown in Quick Start. It verifies checksums, resumes interrupted downloads, extracts HAPS 2.0, and includes the public GitHub annotations required by the simulator.

For the complete HA-R2R training inputs or individual HF components, use the HF CLI in your host Python environment:

pip install huggingface-hub
hf download fly1113/HA-VLN --repo-type dataset --local-dir Data
# Or select a component:
# hf download fly1113/HA-VLN --repo-type dataset --include "HA-R2R/*" --local-dir Data
# hf download fly1113/HA-VLN --repo-type dataset --include "checkpoints/*" --local-dir Data

HF hosts HA-R2R episodes, the HAPS archive, and the CMA checkpoint. HAPS is an archive, not extracted GLBs; the Quick Start downloader handles extraction. Human annotations, collision baselines, and word embeddings are included in this GitHub repository. For training policies from scratch rather than evaluating the released CMA checkpoint, download the pretrained PointGoal ResNet depth observation weights directly from ddppo-models.zip and extract them to Data/ddppo-models/{model}.pth (default: gibson-2plus-resnet50.pth).

bash scripts/download_data.sh

• Data
  • HA-R2R
    • train
    • val_seen
    • val_unseen
  • HAPS2_0
    • balcony:A_child_excitedly_greeting_a_pet._0
    • balcony:A_couple_having_a_quiet,_intimate_conversation._0
    • ......
  • Multi-Human-Annotations
    • human_motion.json
  • HA-R2R-tools
  • ddppo-models
  • scene_datasets

Human Rendering is defined in the class HAVLNCE of HASimulator/environments.py.

Human Rendering uses child threads for timing and the main thread for adding / removing human models and recalculating the required navmesh in real time.

In the first use, the navmesh will be automatically calculated and saved to support operations such as collision calculation, and the subsequent use will directly load the previously generated navmesh. To enable human rendering, modify the following settings in HAVLN-CE task config:

SIMULATOR:
  ADD_HUMAN: True
  HUMAN_GLB_PATH: ../Data/HAPS2_0
  HUMAN_INFO_PATH: ../Data/Multi-Human-Annotations/human_motion.json
  RECOMPUTE_NAVMESH_PATH: ../Data/recompute_navmesh

Human Counting API (Optional): Online human detection and counting relies on GroundingDINO, which is disabled by default (HUMAN_COUNTING: False) and not pre-installed in the Docker image. Standard navigation baseline models (such as HA-VLN-CMA) do not require this module.

If you wish to enable human counting:

1. Set HUMAN_COUNTING: True in HASimulator/config/HAVLNCE_task.yaml (or pass TASK_CONFIG.SIMULATOR.HUMAN_COUNTING True via CLI options).
2. Install GroundingDINO and download its weights by following the setup instructions in Native Installation (for Docker, use only the Python 3.8 optional GroundingDINO block).

We use nine cameras to annotate any anomalies, such as levitation or model clipping, in humans added to the scene. Check details in scripts/human_scene_fusion.py.

• Inspired by: Real-world 3D skeleton tracking techniques.
• Setup:
  • 9 RGB cameras surround each human model to refine position & orientation.
  • Multi-view capture to correct clipping issues with surrounding objects.
• Camera Angles:
  • 8 side cameras: $\theta_{\text{lr}}^{i} = \frac{\pi i}{8}$, alternate up/down tilt.
  • 1 overhead camera: $\theta_{\text{ud}}^{9} = \frac{\pi}{2}$.

To reproduce the Multi-view human annotation videos, run the following script:

cd scripts
python3 human_scene_fusion.py

To modify the output data path, change the following line in scripts/human_scene_fusion.py, or the results will be output in "scripts/test" by default.

output_path = "test/"

You can navigate through a scene with keyboard:

cd scripts
python demo.py --scan 1LXtFkjw3qL

To implement the HA-VLN-CMA agent, you can use the following script:

The instruction embedding notes describe the word-aligned initializer for new training and checkpoint restoration for the released CMA baseline.

cd agent
# Training
python run.py --exp-config config/cma_pm_da_aug_tune.yaml --run-type train

# Evaluation
python run.py --exp-config config/cma_pm_da_aug_tune.yaml --run-type eval

# Inference
python run.py --exp-config config/cma_pm_da_aug_tune.yaml --run-type inference

We present several annotated instances of human subjects from the proposed HAPS 2.0 Dataset (Overall and single), showcasing a variety of well-aligned motions, movements, and interations.

Overall View of Nine Annoated Scenarios from HA-VLN Simulator (90 scans in total)

Single Humans with Movements (910 Humans in total)

Demo 1	Demo 2	Demo 3

Demo 4	Demo 5	Demo 6

🚀🚀🚀 Download Here

Details

Instruction Examples Table presents four instruction examples from the Human-Aware Room-to-Room (HA-R2R) dataset. These cases include various scenarios such as:

• Multi-human interactions (e.g., 1, 2, 3),
• Agent-human interactions (e.g., 1, 2, 3),
• Agent encounters four or more humans (e.g., 3),
• No humans encountered (e.g., 4).

These examples illustrate the diversity of human-aligned navigation instructions that challenge the agent in our task.

Instruction Examples Table

Instruction Example
1. Exit the library and turn left. As you proceed straight ahead, you will enter the bedroom, where you can observe a person actively searching for a lost item, perhaps checking under the bed or inside drawers. Continue moving forward, ensuring you do not disturb his search. As you pass by, you might see a family engaged in a casual conversation on the porch or terrace, be careful not to bump into them. Maintain your course until you reach the closet. Stop just outside the closet and await further instructions.
2. Begin your path on the left side of the dining room, where a group of friends is gathered around a table, enjoying dinner and exchanging stories with laughter. As you move across this area, be cautious not to disturb their gathering. The dining room features a large table and chairs. Proceed through the doorway that leads out of the dining room. Upon entering the hallway, continue straight and then make a left turn. As you walk down this corridor, you might notice framed pictures along the walls. The sound of laughter and conversation from the dining room may still be audible as you move further away. Continue down the hallway until you reach the entrance of the office. Here, you will observe a person engaged in taking photographs, likely focusing on capturing the view from a window or an interesting aspect of the room. Stop at this point, ensuring you are positioned at the entrance without obstructing the photographer's activity.
3. Starting in the living room, you can observe an individual practicing dance moves, possibly trying out new steps. As you proceed straight ahead, you will pass by couches where a couple is engaged in a quiet, intimate conversation, speaking softly to maintain their privacy. Continue moving forward, ensuring you navigate around any furniture or obstacles in your path. As you transition into the hallway, notice another couple enjoying a date night at the bar, perhaps sharing drinks and laughter. Maintain a steady course without disturbing them, keeping to the right side of the hallway. Upon reaching the end of your path, you will find yourself back in the living room. Here, a person is checking their appearance in a hallway mirror, possibly adjusting their attire or hair. Stop by the right candle mounted on the wall, ensuring you are positioned without blocking any pathways.
4. Begin by leaving the room and turning to your right. Proceed down the hallway, be careful of any human activity or objects along the way. As you continue, look for the first doorway on your right. Enter through this doorway and advance towards the shelves. Once you reach the vicinity of the shelves, come to a halt and wait there. During this movement, avoid any obstacles or disruptions in the environment.

purple-highlighted instructions relate to human movements, and blue-highlighted instructions are associated with agent-human interactions.

---

Details

🌍 Stage 1: Coarse Annotation

• Goal: Assign human motions to specific regions and objects using a coarse-to-fine approach.
• Process:
  • Filter human motions $\mathbf{H}$ based on region $\mathbf{R}$ and object list $\mathbf{O}$.
  • Match motions $h_i$ with objects $j_i$ using semantic similarity.
  • Optimize human placements $\mathbf{p}_{opt}^{h_i}$ using Particle Swarm Optimization (PSO).
• Constraints:
  • Search space limited by region boundaries.
  • Maintain minimum safe distance $\epsilon = 1m$ from other objects.
  • Ensures naturalistic human placements for training navigation agents.

🎥 Stage 2: Fine Annotation

• Inspired by: Real-world 3D skeleton tracking techniques.
• Setup:
  • 9 RGB cameras surround each human model to refine position & orientation.
  • Multi-view capture to correct clipping issues with surrounding objects.
• Camera Angles:
  • 8 side cameras: $\theta_{\text{lr}}^{i} = \frac{\pi i}{8}$, alternate up/down tilt.
  • 1 overhead camera: $\theta_{\text{ud}}^{9} = \frac{\pi}{2}$.
• Scale: 529 human models annotated in 374 regions across 90 scans.

👥 Multi-Human Interaction & Motion Enrichment

• Goal: Increase scene diversity and human interactions.
• Process:
  • Use LLMs to generate new multi-human interactions.
  • Manual refinement (4 rounds) ensures consistency.
  • Place new motions relative to objects & use multi-camera annotation.
• Result:
  • 910 human models across 428 regions.
  • Complex motions: Walking downstairs, climbing stairs.
  • Interaction stats: 72 two-human pairs, 59 three-human pairs, 15 four-human groups.
• Impact: Enables precise social modeling for human-aware navigation.

We welcome contributions to this project! Please contact yfeidong@uw.edu , fyiwu@uw.edu , or bohanx2@uw.edu.

If you find this repository or our paper useful, please consider starring this repository and citing our paper:

@inproceedings{dong2026havln,
  title={{HA-VLN 2.0: An Open Benchmark and Leaderboard for Human-Aware Navigation in Discrete and Continuous Environments with Dynamic Multi-Human Interactions}},
  author={Dong, Yifei and Wu, Fengyi and He, Qi and Kong, Lingdong and Li, Heng and Li, Minghan and Cheng, Zebang and Zhou, Yuxuan and Sun, Jingdong and Dai, Qi and Hauptmann, Alexander G. and Cheng, Zhi-Qi},
  booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year={2026},
  url={https://arxiv.org/abs/2503.14229}
}

This project is licensed under the MIT License. For more details, see the LICENSE file.