JIT

Scaling Harness Intelligence via Just-in-Time Harness Evolution

What is JIT-Agent?

JIT-Agent is a compact meta-agent that writes your agent harness on the fly. Instead of precompiling one general-purpose scaffold and hoping it transfers, JIT-Agent takes a task spec, a protocol, a tool/skill registry, and a few retrieved prior harnesses, and emits an executable, task-specific harness that wraps any off-the-shelf agentic LLM — Model-as-a-Harness.

Overview of JIT-Agent. Given a task, JIT-Agent composes a problem-specific agent harness by selecting and instantiating four modules: memory, planning, action, and capability. Different task structures therefore induce distinct executable protocols and state organizations.

Every harness is factored into four modules — memory, planning, action, capability orchestration — implemented against the shared interfaces in HarnessFactory, so generation means emitting structured code rather than free-form agent programs. As traces and feedback come back, JIT-Agent revises the harness and updates the archive: harnesses keep improving at test time while the generator itself stays frozen.

Results. The resulting JIT-Agent-27B lifts a wide range of backbone agents across deep research, daily work, planning, and workspace tasks.

Building the scaffold turns out to be a trainable, transferable axis of agent intelligence — orthogonal to scaling the base model.

Repository layout

Each directory has its own README with the details.

Setup

1. Clone the repository

git clone https://github.com/bingreeky/JIT.git
cd JIT

2. Environment (Python 3.11)

conda env create -f environment.yml && conda activate jit

or, in an existing environment: pip install -r requirements.txt. Serving a local meta model (vLLM/SGLang + torch) is deliberately not included — the pipeline only ever talks HTTP to it.

3. Credentials

cp .env.example .env   # then fill it in

Anything already exported in the shell wins over .env, and every model role can also be overridden per run on the command line.

4. Data

Small datasets ship in the repo; anything large is a documented download.

python scripts/check_datasets.py        # present / partial / missing, per benchmark
bash scripts/fetch_datasets.sh travel   # one benchmark ("all" ≈ 1 GB)

Usage

All modes share the same benchmark adapters, execution model, judge, and scoring path. The meta model writes a harness, the execution model runs it, and the judge model grades the result. Configure credentials in .env; CLI flags override them.

1. Test a fixed HarnessFactory design. No meta model is called; the selected harness is executed and scored directly.

python -m scripts.run_seed_harness --bench xbench --list-harnesses
python -m scripts.run_seed_harness --bench xbench \
    --harness plan_and_execute --max-samples 5

See the HarnessFactory guide for the eleven included designs.

2. Use a hosted API model as the meta-agent. Hosted APIs usually do not expose prompt_logprobs, so use judge selection explicitly. META_API_KEY is read from .env.

python -m scripts.run_jit --bench xbench \
    --meta-model provider-model --meta-base https://api.provider.com/v1 \
    --selector judge --rollouts 3 --max-samples 5

3. Evaluate the JIT checkpoint. Serve the checkpoint, then use its tokenizer for the published log-probability selector.

MODEL=JIT-Agent/jit-27b SERVED_NAME=jit TP=4 \
    bash scripts/serve_meta_model.sh

python -m scripts.run_jit --bench xbench \
    --meta-model jit --meta-base http://127.0.0.1:8000/v1 \
    --selector logprob --tokenizer JIT-Agent/jit-27b \
    --rollouts 3 --meta-temperature 1.0 --max-samples 5

Drop --max-samples for a full run. MODEL may also be a local checkpoint path; SERVED_NAME must match --meta-model. The shell wrapper bash scripts/run_jit.sh xbench reads the same settings from the environment.

Key arguments

Supported benchmarks are xbench, deepsearchqa, agentif, officebench, odyssey, shopping, and travel.

Output and resume. JIT runs separate generation, selection, and execution artifacts:

summary.json    headline metrics + how the run was configured
generate/       the N candidate harnesses per case, with prompts and responses
select/         the pick per case, the rule that produced it, per-candidate scores
execute/        the harness that actually ran, its trajectory, and the numbers you report

Fixed-harness runs write summary.json, scores.jsonl, and per-case reports directly. Re-running an identical command resumes completed work and retries only infrastructure failures; --skip-generate and --skip-select reuse earlier JIT phases.

Detailed documentation: JIT pipeline · HarnessFactory · CLI and runtime. Run either entry point with --help for the full flag list.

Citation

If you find JIT-Agent useful, please cite:

@misc{zhang2026jitagentscalingharnessintelligence,
      title={JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution},
      author={Guibin Zhang and Leo Lu and Fangzhou Xie and Kang Zhu and Junhao Wang and Zhifei Xie and Zhaochen Yu and Zihang Liu and Zhongxiang Sun and Qiankun Li and Yue Liao and Heng Chang and Xiaobin Hu and Qibing Ren and Wangchunshu Zhou and Shuicheng Yan},
      year={2026},
      eprint={2608.25593},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2608.25593},
}

License

See LICENSE.