AI Rank
中文
← Open Source
mees

calvin

CALVIN - A benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks.

Model DevelopmentEnvironmentsReinforcement LearningPython
Open on GitHub
Momentum
+1stars in 24 hours+0.1%
998
Stars
127
Forks
+2
This week
5
Contributors
Created 2021-07-20 · Updated 2026-10-16 · #5950 today
Top developers
meeslukashermannErickRosete

mees/calvin is ranked 5,950 with 998 stars.

Same-category comparison

mees/calvin is ranked 5,950 among these 8 peers; havakv/pycox is ranked 5,947.

Same-category comparison
NameRankMetrics
LeCAR-Lab/dial-mpc5,932
Language:
Python
Stars:
1,011
Forks:
104
Description:
Official implementation for the paper "Full-Order Sampling-Based MPC for Torque-Level Locomotion Control via Diffusion-Style Annealing". DIAL-MPC is a novel sampling-based MPC framework for legged robot full-order torque-level control with both precision and agility in a training-free manner.
TinyLLaVA/TinyLLaVA_Factory5,935
Language:
Python
Stars:
1,010
Forks:
100
Description:
A Framework of Small-scale Large Multimodal Models.
pjlab-sys4nlp/llama-moe5,940
Language:
Python
Stars:
1,005
Forks:
60
Description:
⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training (EMNLP 2024).
havakv/pycox5,947
Language:
Python
Stars:
1,000
Forks:
205
Description:
Survival analysis with PyTorch.
zhouhaoyi/ETDataset5,957
Stars:
991
Forks:
181
Description:
The Electricity Transformer dataset is collected to support the further investigation on the long sequence forecasting problem.
qiuzh20/gated_attention5,961
Language:
Jupyter Notebook
Stars:
987
Forks:
61
Description:
The official implementation for [NeurIPS2025 Oral] Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free.
lucidrains/BS-RoFormer5,967
Language:
Python
Stars:
984
Forks:
49
Description:
Implementation of Band Split Roformer, SOTA Attention network for music source separation out of ByteDance AI Labs.
nicklashansen/tdmpc25,973
Language:
Python
Stars:
981
Forks:
212
Description:
Code for "TD-MPC2: Scalable, Robust World Models for Continuous Control".
Key parameters and specifications
Key parameters and specifications
ParameterValue
LanguagePython
CategoryModel Development
SubcategoryEnvironments, Reinforcement Learning
Stars998
Forks127
Contributors5
Created date2021-07-20
Updated date2026-10-16
Stars in 1 day1
Stars in 7 days2
Rank5,950
Previous rank12,084
Rank change6,134
7-day stars rank8,314
Previous 7-day stars rank10,369
7-day stars rank change2,055
Total stars rank11,162
Previous total stars rank11,153
Total stars rank change-9
CountryUSA
README summary

CALVIN CALVIN - A benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks Oier Mees, Lukas Hermann, Erick Rosete, Wolfram Burgard CALVIN won the 2022 IEEE Robotics and Automation Letters (RA-L) Best Paper Award! We present **CALVIN** (**C**omposing **A**ctions from **L**anguage and **Vi**sio**n**), an open-source simulated benchmark to learn long-horizon language-conditioned tasks. Our aim is to make it possible to develop agents that can solve many robotic manipulation tasks

over a long horizon, from onboard sensors, and specified only via human language. CALVIN tasks are more complex in terms of sequence length, action space, and language than existing vision-and-language task datasets and supports flexible specification of sensor suites. :computer: Quick Start To begin, clone this repository locally Install requirements: If you encounter problems installing pyhash, you might have to downgrade setuptools to a version below 58. Download dataset (choose which split you want to download with the

argument , or ): \ If you want to get started without downloading the whole dataset, use the argument to download a small debug dataset (1. :weight_lifting_man: Train Baseline Agent Train baseline models: The option loads the CALVIN dataset into shared memory at the beginning of the training, speeding up the data loading during training. The preparation of the shared memory cache will take some time (approx. \ If you want to use the original data loader (e. for debugging) just override

the command with . \ For an additional speed up, you can disable the evaluation callbacks during training by adding and You want to scale your training to a multi-gpu setup? Just specify the number of GPUs and DDP will automatically be used for training thanks to Pytorch Lightning. To train on all available GPUs: If you have access to a Slurm cluster, follow this guide. You can use Hydra's flexible overriding system for changing hyperparameters. For example, to train a

model with rgb images from both static camera and the gripper camera with relative actions: To train a model with RGB-D from both cameras: To train a model with rgb images from the static camera and visual tactile observations with absolute actions: To see all available hyperparameters: To resume a training, just override the hydra working directory : :framed_picture: Sensory Observations CALVIN supports a range of sensors commonly utilized for visuomotor control: 1. **Static camera RGB images** - with shape

. **Static camera Depth maps** - with shape . **Gripper camera RGB images** - with shape . **Gripper camera Depth maps** - with shape . **Proprioceptive state** - EE position (3), EE orientation in euler angles (3), gripper width (1), joint positions (7), gripper action (1). :joystick: Action Space In CALVIN, the agent must perform closed-loop continuous control to follow unconstrained language instructions characterizing complex robot manipulation tasks, sending continuous actions to the robot at 30hz. In order to give researchers and practitioners the freedom to experiment

with different action spaces, CALVIN supports the following actions spaces: 1. **Absolute cartesian pose** - EE position (3), EE orientation in

Frequently asked questions

How many stars and forks are recorded?

mees/calvin has 998 stars and 127 forks.

What is the repository rank?

mees/calvin is ranked 5,950.

What language and category are recorded?

mees/calvin has 998 stars, uses Python and belongs to Model Development.

Which repository is closest in rank?

mees/calvin is compared with havakv/pycox, ranked 5,947 with 1,000 stars.

README

CALVIN

CALVIN - A benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks

Oier Mees, Lukas Hermann, Erick Rosete, Wolfram Burgard

CALVIN won the 2022 IEEE Robotics and Automation Letters (RA-L) Best Paper Award!

We present CALVIN (Composing Actions from Language and Vision), an open-source simulated benchmark to learn long-horizon language-conditioned tasks. Our aim is to make it possible to develop agents that can solve many robotic manipulation tasks over a long horizon, from onboard sensors, and specified only via human language. CALVIN tasks are more complex in terms of sequence length, action space, and language than existing vision-and-language task datasets and supports flexible specification of sensor suites.

:computer: Quick Start

To begin, clone this repository locally

git clone --recurse-submodules https://github.com/mees/calvin.git
$ export CALVIN_ROOT=$(pwd)/calvin

Install requirements:

$ cd $CALVIN_ROOT
$ conda create -n calvin_venv python=3.8  # or use virtualenv
$ conda activate calvin_venv
$ sh install.sh

If you encounter problems installing pyhash, you might have to downgrade setuptools to a version below 58.

Download dataset (choose which split you want to download with the argument D, ABC or ABCD):
If you want to get started without downloading the whole dataset, use the argument debug to download a small debug dataset (1.3 GB).

$ cd $CALVIN_ROOT/dataset
$ sh download_data.sh D | ABC | ABCD | debug

:weight_lifting_man: Train Baseline Agent

Train baseline models:

$ cd $CALVIN_ROOT/calvin_models/calvin_agent
$ python training.py datamodule.root_data_dir=/path/to/dataset/ datamodule/datasets=vision_lang_shm

The vision_lang_shm option loads the CALVIN dataset into shared memory at the beginning of the training, speeding up the data loading during training. The preparation of the shared memory cache will take some time (approx. 20 min at our SLURM cluster).
If you want to use the original data loader (e.g. for debugging) just override the command with datamodule/datasets=vision_lang.
For an additional speed up, you can disable the evaluation callbacks during training by adding ~callbacks/rollout and ~callbacks/rollout_lh

You want to scale your training to a multi-gpu setup? Just specify the number of GPUs and DDP will automatically be used for training thanks to Pytorch Lightning. To train on all available GPUs:

$ python training.py trainer.gpus=-1

If you have access to a Slurm cluster, follow this guide.

You can use Hydra's flexible overriding system for changing hyperparameters. For example, to train a model with rgb images from both static camera and the gripper camera with relative actions:

$ python training.py datamodule/observation_space=lang_rgb_static_gripper_rel_act model/perceptual_encoder=gripper_cam

To train a model with RGB-D from both cameras:

$ python training.py datamodule/observation_space=lang_rgbd_both model/perceptual_encoder=RGBD_both

To train a model with rgb images from the static camera and visual tactile observations with absolute actions:

$ python training.py datamodule/observation_space=lang_rgb_static_tactile_abs_act model/perceptual_encoder=static_RGB_tactile

To see all available hyperparameters:

$ python training.py --help

To resume a training, just override the hydra working directory :

$ python training.py hydra.run.dir=runs/my_dir

:framed_picture: Sensory Observations

CALVIN supports a range of sensors commonly utilized for visuomotor control:

  1. Static camera RGB images - with shape 200x200x3.
  2. Static camera Depth maps - with shape 200x200.
  3. Gripper camera RGB images - with shape 84x84x3.
  4. Gripper camera Depth maps - with shape 84x84.
  5. Tactile image - with shape 120x160x6.
  6. Proprioceptive state - EE position (3), EE orientation in euler angles (3), gripper width (1), joint positions (7), gripper action (1).

:joystick: Action Space

In CALVIN, the agent must perform closed-loop continuous control to follow unconstrained language instructions characterizing complex robot manipulation tasks, sending continuous actions to the robot at 30hz. In order to give researchers and practitioners the freedom to experiment with different action spaces, CALVIN supports the following actions spaces:

  1. Absolute cartesian pose - EE position (3), EE orientation in euler angles (3), gripper action (1).
  2. Relative cartesian displacement - EE position (3), EE orientation in euler angles (3), gripper action (1).
  3. Joint action - Joint positions (7), gripper action (1).

For more information, please refer to this more detailed README.

:muscle: Evaluation: The Calvin Challenge

Long-horizon Multi-task Language Control (LH-MTLC)

The aim of the CALVIN benchmark is to evaluate the learning of long-horizon language-conditioned continuous control policies. In this setting, a single agent must solve complex manipulation tasks by understanding a series of unconstrained language expressions in a row, e.g., “open the drawer. . . pick up the blue block. . . now push the block into the drawer. . . now open the sliding door”. We provide an evaluation protocol with evaluation modes of varying difficulty by choosing different combinations of sensor suites and amounts of training environments. To avoid a biased initial position, the robot is reset to a neutral position before every multi-step sequence.

To evaluate a trained calvin baseline agent, run the following command:

$ cd $CALVIN_ROOT/calvin_models/calvin_agent
$ python evaluation/evaluate_policy.py --dataset_path  --train_folder

Optional arguments:

  • --checkpoint : by default, the evaluation loads the last checkpoint in the training log directory. You can instead specify the path to another checkpoint by adding this to the evaluation command.
  • --debug: print debug information and visualize environment.

If you want to evaluate your own model architecture on the CALVIN challenge, you can implement the CustomModel class in evaluate_policy.py as an interface to your agent. You need to implement the following methods:

  • __init__(): gets called once at the beginning of the evaluation.
  • reset(): gets called at the beginning of each evaluation sequence.
  • step(obs, goal): gets called every step and returns the predicted action.

Then evaluate the model by running:

$ python evaluation/evaluate_policy.py --dataset_path  --custom_model

You are also free to use your own language model instead of using the precomputed language embeddings provided by CALVIN. For this, implement CustomLangEmbeddings in evaluate_policy.py and add --custom_lang_embeddings to the evaluation command.

Multi-task Language Control (MTLC)

Alternatively, you can evaluate the policy on single tasks and without resetting the robot to a neutral position. Note that this evaluation is currently only available for our baseline agent.

$ python evaluation/evaluate_policy_singlestep.py --dataset_path  --train_folder  [--checkpoint ] [--debug]

Pre-trained Model

Download the MCIL model checkpoint trained on the static camera rgb images on environment D.

$ wget http://calvin.cs.uni-freiburg.de/model_weights/D_D_static_rgb_baseline.zip
$ unzip D_D_static_rgb_baseline.zip

:speech_balloon: Relabeling Raw Language Annotations

You want to try learning language conditioned policies in CALVIN with a new awesome language model?

We provide an example script to relabel the annotations with different language model provided in SBert, such as the larger MPNet (paraphrase-mpnet-base-v2) or its corresponding multilingual model (paraphrase-multilingual-mpnet-base-v2). The supported options are "mini", "mpnet" and "multi". If you want to try different SBert models, just change the model name here.

cd $CALVIN_ROOT/calvin_models/calvin_agent
python utils/relabel_with_new_lang_model.py +path=$CALVIN_ROOT/dataset/task_D_D/ +name_folder=new_lang_model_folder model.nlp_model=mpnet

If you additionally want to sample different language annotations for each sequence (from the same task annotations) in the training split run the same command with the parameter reannotate=true.

:chart_with_upwards_trend: SOTA Models

Open-source models that outperform the MCIL baselines from CALVIN:

For a detailed overview of the evaluation performances, have a look at our LEADERBOARD.

Grounding Language with Visual Affordances over Unstructured Data

Oier Mees, Jessica Borja-Diaz, Wolfram Burgard

Paper, Code

FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policies

Moritz Reuss, Hongyi Zhou, Marcel Rühle, Ömer Erdinç Yağmurlu, Fabian Otto, Rudolf Lioutikov

Paper, Code

Unified Vision-Language-Action Model

Yuqi Wang, Xinghang Li, Wenxuan Wang, Junbo Zhang, Yingyan Li, Yuntao Chen, Xinlong Wang, Zhaoxiang Zhang

Paper, Code

Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation

Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, Jiangmiao Pang

Paper, Code

Diffusion Transformer Policy: Scaling Diffusion Transformer for Generalist Vision-Language-Action Learning

Zhi Hou, Tianyi Zhang, Yuwen Xiong, Hengjun Pu, Chengyang Zhao, Ronglei Tong, Yu Qiao, Jifeng Dai, Yuntao Chen

Paper, Code

GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal Conditioned Policy

Peiyan Li, Hongtao Wu, Yan Huang, Chilam Cheang, Liang Wang, Tao Kong

Paper, Code

GHIL-Glue: Hierarchical Control with Filtered Subgoal Images

Kyle B Hatch, Ashwin Balakrishna, Oier Mees, Suraj Nair, Seohong Park, Blake Wulfe, Masha Itkina, Benjamin Eysenbach, Sergey Levine, Thomas Kollar, Benjamin Burchfiel

Paper, Code

Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learning

Moritz Reuss, Jyothish Pari, Pulkit Agrawal, Rudolf Lioutikov

Paper, Code

Incorporating Task Progress Knowledge for Subgoal Generation in Robotic Manipulation through Image Edits

Xuhui Kang, Yen-Ling Kuo

Paper, Code

Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation

Qingwen Bu, Jia Zeng, Li Chen, Yanchao Yang, Guyue Zhou, Junchi Yan, Ping Luo, Heming Cui, Yi Ma, Hongyang Li

Paper, Code

DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution

Yang Yue, Yulin Wang, Bingyi Kang, Yizeng Han, Shenzhi Wang, Shiji Song, Jiashi Feng, Gao Huang

Paper, Code

RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulation

Fanfan Liu, Feng Yan, Liming Zheng, Yiyang Huang, Chengjian Feng, Lin Ma

Paper, Code

Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goals

Moritz Reuss, Ömer Erdinç Yağmurlu, Fabian Wenzel, Rudolf Lioutikov

Paper, Code

3D Diffuser Actor: Policy Diffusion with 3D Scene Representations

Tsung-Wei Ke, Nikolaos Gkanatsios, Katerina Fragkiadaki

Paper, Code

Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation

Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, Tao Kong

Paper, Code

Vision-Language Foundation Models as Effective Robot Imitators

Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, Hang Li, and Tao Kong

Paper, Code

Zero-Shot Robotic Manipulation With Pretrained Image-Editing Diffusion Models

Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, Sergey Levine

Paper, Code

Language Control Diffusion: Efficiently Scaling through Space, Time, and Tasks

Eddie Zhang, Yujie Lu, William Wang, Amy Zhang

Paper, Code

What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data

Oier Mees, Lukas Hermann, Wolfram Burgard

Paper, Code

Language-Conditioned Imitation Learning with Base Skill Priors under Unstructured Data

Hongkuan Zhou, Zhenshan Bing, Xiangtong Yao, Xiaojie Su, Chenguang Yang, Kai Huang, Alios Knoll

Paper, Code

Contact Oier to add your model here.

Reinforcement Learning with CALVIN

Are you interested in trying reinforcement learning agents for the different manipulation tasks in the CALVIN environment? We provide a google colab to showcase how to leverage the CALVIN task indicators to learn RL agents with a sparse reward.

FAQ

Why do you use EGL rendering?

We use EGL to move the bullet rendering from cpu (which is the default) to gpu, which is much faster. This way, we can also do rollouts during the training of the agent to track its performance. By changing from cpu to gpu, the rendered textures change slightly, so be aware of this if you plan on testing pretrained models.

I am training with multiple GPUs and why am I get OOM errors during rollouts?

PyBullet only recently added an option to select which GPU to use for rendering when using EGL (fix was commited in 3c4cb80 on Oct 22, 2021, see here. If you have an old version of PyBullet, there is no way to choose the GPU, which can lead to problems on cluster nodes with multiple GPUs, because all instances would be placed on the same GPU, slowing down the rendering and potentially leading to OOM erros.

The fix introduced an environment variable EGL_VISIBLE_DEVICES (similar to CUDA_VISIBLE_DEVICES) which lets you specify the GPU device to render on. However, there is one catch: On some machines, the device ids of CUDA and EGL do not match (e.g. CUDA device 0 could be EGL device 3). We automatically handle this in our wrapper in calvin_env and find the corresponding egl device id, so you don't have to set EGL_VISIBLE_DEVICES yourself, see here.

I am not interested in the manipulation tasks recorded, can I record different demonstration with teleop?

Yes, although it is not documented right now, all the code to record data with a VR headset is present in calvin_env in https://github.com/mees/calvin_env/blob/main/calvin_env/vrdatacollector.py

Changelog

24 Feb 2023

  • Wrong scene_info.npy in D dataset. Note that we have updated the corresponding checksum. Please replace as follows:
cd task_D_D
wget http://calvin.cs.uni-freiburg.de/scene_info_fix/task_D_D_scene_info.zip
unzip task_D_D_scene_info.zip && rm task_D_D_scene_info.zip

16 Sep 2022

  • MAJOR BUG IN ABC and ABCD dataset: If you downloaded these datasets before this date you have to do these fixes:
    • Wrong language annotations in ABC and ABCD dataset. You can download the corrected language embeddings here.
    • Bug in calvin_env that only affects the generation of language embeddings.
    • Wrong scene_info.npy in ABC and ABCD dataset. Please replace as follows:
cd task_ABCD_D
wget http://calvin.cs.uni-freiburg.de/scene_info_fix/task_ABCD_D_scene_info.zip
unzip task_ABCD_D_scene_info.zip && rm task_ABCD_D_scene_info.zip

cd task_ABC_D
wget http://calvin.cs.uni-freiburg.de/scene_info_fix/task_ABC_D_scene_info.zip
unzip task_ABC_D_scene_info.zip && rm task_ABC_D_scene_info.zip
  • Added additional language embeddings to dataset.

15 May 2022

  • Added shared memory dataset loader for faster training. Refactored data loading classes.

7 Feb 2022

  • Minor changes to the distribution of tasks in the long-horizon multi-step sequences.
  • Changes to the task success criteria of pushing and lifting.
  • Set use_nullspace: true for robot in hydra cfg of dataset. If you downloaded one of the datasets prior to this date, edit this line in /training/.hydra/merged_config.yaml and /validation/.hydra/merged_config.yaml.
  • Renaming model.decoder to model.action_decoder.

10 Jan 2022

  • Breaking change to evaluation, using different intitial states for environment.

Citation

If you find the dataset or code useful, please cite:

@article{mees2022calvin,
author = {Oier Mees and Lukas Hermann and Erick Rosete-Beas and Wolfram Burgard},
title = {CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks},
journal={IEEE Robotics and Automation Letters (RA-L)},
volume={7},
number={3},
pages={7327-7334},
year={2022}
}

License

MIT License

Nearby in the ranking
Similar items