Meta Agents Research Environments (ARE)

A research environment for simulating complex, real-life tasks that require multi-step reasoning and dynamic adaptation.

Meta Agents Research Environments (ARE) is a platform designed to evaluate AI agents in dynamic, realistic scenarios. Unlike static benchmarks, this research platform introduces evolving environments where agents must adapt their strategies as new information becomes available, mirroring real-world challenges. In particular, ARE runs the Gaia2 benchmark, a follow-up to Gaia, evaluating a broader range of agent capabilities.

Looking for the container-based Gaia2-CLI benchmark stack? Start with gaia2-cli.

New: facebook/omnilingual-gaia2, a multilingual extension of Gaia2! See Omnilingual-Gaia2 below.

Table of Contents

• Background
• Install
• Usage
• API
• Contributing
• License

Background

ARE addresses critical gaps in AI agent evaluation by providing:

• Dynamic Environments: Scenarios that evolve over time with new information and changing conditions
• Multi-Step Reasoning: Complex tasks requiring 10+ steps and several minutes to complete
• Real-World Focus: Grounded situations that mirror actual real-world challenges
• Comprehensive Evaluation: The Gaia2 benchmark with 800 scenarios across multiple domains

Getting Started

Install

For complete installation instructions and setup options, see the Installation Guide.

Prerequisites

First, install uv, a fast Python package installer and resolver.

Quick Start with uvx

The fastest way to get started is using uvx to run commands directly:

# Run Gaia2 benchmark scenarios
uvx --from meta-agents-research-environments are-benchmark gaia2-run --hf meta-agents-research-environments/gaia2 --hf_split validation -l 1

# Run custom scenarios
uvx --from meta-agents-research-environments are-run -s scenario_tutorial -a default

All the commands in this README and the documentation are available through uvx.

Traditional Installation

Alternatively, install the package directly with different dependency sets:

# Minimal install (core dependencies only)
# Good for basic benchmarking and running most scenarios
pip install meta-agents-research-environments

# With GUI (recommended for interactive exploration)
pip install "meta-agents-research-environments[gui]"

Which installation should I choose?

• Minimal (meta-agents-research-environments): For running benchmarks and scenarios via CLI
• With GUI ([gui]): Adds web interface for interactive exploration (recommended for local development)

Usage

Basic Commands

After installation, these command-line tools are available:

Run Individual Scenarios

are-run -s scenario_find_image_file -a default

Benchmark Evaluation

are-benchmark run -d /path/to/scenarios --agent default --limit 10

Gaia2 Evaluation

are-benchmark gaia2-run --hf meta-agents-research-environments/gaia2 --hf_split validation -l 5

Interactive GUI

are-gui -s scenario_find_image_file

The GUI provides a web-based interface for interactive scenario exploration and real-time agent monitoring. When started, it typically runs at http://localhost:8080. The interface supports different view modes:

• Playground Mode: Chat-like interface for direct agent interaction
• Scenarios Mode: Structured task execution and evaluation with DAG visualization

For detailed information about the GUI features, navigation, and workspace usage, see the Understanding UI Guide.

Model Configuration

ARE supports multiple AI model providers through LiteLLM:

# Llama API
export LLAMA_API_KEY="your-api-key"
are-benchmark run --hf meta-agents-research-environments/gaia2 --hf_split validation \
  --model Llama-3.1-70B-Instruct --provider llama-api --agent default

# Local deployment
are-benchmark run --hf meta-agents-research-environments/gaia2 --hf_split validation \
  --model your-local-model --provider local \
  --endpoint "http://localhost:8000" --agent default

For detailed information on configuring different model providers, environment variables, and advanced options, see the LLM Configuration Guide.

Run any command with --help to see all available options.

Example: Gaia2 Benchmark

# Set up your model configuration
export LLAMA_API_KEY="your-api-key"

# Run a validation set to test your setup
are-benchmark run --hf meta-agents-research-environments/gaia2 --hf_split validation \
  --model meta-llama/Llama-3.3-70B-Instruct --model_provider novita \
  --agent default --limit 10 --output_dir ./validation_results

# Run complete Gaia2 evaluation for leaderboard submission
are-benchmark gaia2-run --hf meta-agents-research-environments/gaia2 \
  --model Llama-3.1-70B-Instruct --provider llama-api \
  --agent default --output_dir ./gaia2_results \
  --hf_upload my-org/gaia2-results

Omnilingual-Gaia2

facebook/omnilingual-gaia2 is a multilingual extension of Gaia2 — 10 languages across the execution, search, ambiguity and adaptability splits. The gaia2-cli runner downloads and caches it for you: set [target].language to pick the language, and [judge].prompt_version = "omnilingual-gaia2" to enable the multilingual judge prompts, without which a correct non-English answer is often scored as a miss.

# Copy the annotated template, then edit the language, models and endpoints
cp gaia2-cli/runner/examples/openclaw_qwen_omnilingual_gaia2_pass3.toml ./my_omnilingual_run.toml

# Validate the config and resolve the scenario selection without launching
export OPENAI_COMPAT_API_KEY="your-api-key"
gaia2-runner run-config --config ./my_omnilingual_run.toml --dry-run

# Run the evaluation
gaia2-runner run-config --config ./my_omnilingual_run.toml

The relevant part of the template:

[target]
dataset = "facebook/omnilingual-gaia2"
language = "spa_Latn"   # or cmn_Hans, deu_Latn, fra_Latn, ...

[judge]
prompt_version = "omnilingual-gaia2"

To translate Gaia2 into another language yourself, see gaia2-cli/mt.

Sequential Gaia2

Sequential Gaia2 chains the scenarios of each universe. Every scenario still runs in a fresh container, but it starts from the agent's memory, conversation and app state left by the previous scenario of its chain. The gaia2-cli runner ships chain orderings for the search (160 scenarios) and execution (95 scenarios) splits, one chain per universe. OpenClaw images only.

# Validate the example config and resolve the chains without launching
export ANTHROPIC_API_KEY="your-api-key"
gaia2-runner run-config --config gaia2-cli/runner/examples/openclaw_sonnet_gaia2_sequential_search.toml --dry-run

# Run the evaluation
gaia2-runner run-config --config gaia2-cli/runner/examples/openclaw_sonnet_gaia2_sequential_search.toml

The example finds its sequence file relative to its own location, so update [target].sequences if you copy it elsewhere. See Sequential Runs for the sequence format, checkpoints, --retry and the context window override.

API

Core Concepts

• Agents: AI entities that interact with the environment using ReAct (Reasoning + Acting) framework
• Apps: Interactive applications (email, calendar, file system) that provide APIs for agent interaction
• Events: Dynamic elements that make environments evolve over time
• Scenarios: Complete tasks combining apps, events, and validation logic

Documentation

Comprehensive documentation is available at:

• Main Documentation: docs/index.rst
• Tutorials: docs/tutorials/
• API Reference: docs/api_reference/

Key documentation sections:

• Core Concepts - Understanding agents, apps, events, and scenarios
• Benchmarking Guide - Complete benchmarking and evaluation reference
• Gaia2 Evaluation - Detailed Gaia2 benchmark submission guide
• Scenario Development - Creating custom scenarios
• CLI Reference - Complete command-line interface documentation

Quick Links

• Installation Guide: docs/user_guide/installation.rst
• Quickstart Tutorial: docs/quickstart.rst

Contributing

We welcome contributions! Please see our Contributing Guide for details on:

• Setting up the development environment
• Running tests and linting
• Submitting pull requests
• Creating new scenarios and apps

License

This project is licensed under the MIT License. See the LICENSE file for details.

Citation

If you use Meta Agents Research Environments in your work, please cite:

@misc{froger2025arescalingagentenvironments,
      title={ARE: Scaling Up Agent Environments and Evaluations},
      author={Romain Froger and Pierre Andrews and Matteo Bettini and Amar Budhiraja and Ricardo Silveira Cabral and Virginie Do and Emilien Garreau and Jean-Baptiste Gaya and Hugo Laurençon and Maxime Lecanu and Kunal Malkan and Dheeraj Mekala and Pierre Ménard and Gerard Moreno-Torres Bertran and Ulyana Piterbarg and Mikhail Plekhanov and Mathieu Rita and Andrey Rusakov and Vladislav Vorotilov and Mengjue Wang and Ian Yu and Amine Benhalloum and Grégoire Mialon and Thomas Scialom},
      year={2025},
      eprint={2509.17158},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2509.17158},
}