vllm-jev

Fast structured decisions on Linux and Apple Silicon

Documentation · Demos · Getting Started · Supported Models · Example · Updates

Demos

🎉 Celebrating RadixFlow! (experimental)

https://github.com/user-attachments/assets/d99f5a8d-d715-4a79-8870-e56a6e48c073

Our first PR, which explored shared-prefix reuse and cache-on/off behavior, inspired us to look more closely at what happens after a cache hit. Reuse matters especially for prefill: Jev scores candidate branches through prefill, so reusing their common prefix avoids repeated computation.

We found a scheduling gap: a matching prefix can still have hybrid state being produced in the current round. Stopping the scan at that pending branch leaves room that independent ready branches could use. RadixFlow defers the pending branch, continues selecting ready work, and rechecks it after completion. The engineering integration recovered executable capacity in the tested hybrid models; the pressure comparisons report the resulting worker rounds, speed and token reuse.

RadixFlow is experimental and disabled by default. Enable it with --radixflow true on supported configurations.

We welcome bug reports, feedback, and discussions—please share your experience with RadixFlow!

Video understanding: live comparison

The same Valen checkpoint answers 24 structured questions about one video, using eight sampled frames and one A800 per runtime. In this live run, Valen's PyTorch reference path took 5.31 s, while vLLM Jev took 1.22 s (4.3× faster); all 24 selected answers agreed. Model loading is excluded. Watch the MP4.

Apple Silicon: live image decisions

Valen 2B answers six GameQA Maze image questions locally on an Apple M5 using MLX. The display follows live HTTP responses. Watch the MP4.

More demos

Multimodal: live Sokoban

Both runtimes played the same five levels from Valen's Sokoban evaluation set with the same checkpoint and one A800 each.

Text: 40 concurrent decisions

The Open-Jev-2B author HTTP server and vLLM Jev received the same 40 text-only Choice requests at once, on one A800 each.

About

vLLM Jev serves compatible Jev-style checkpoints through vLLM. Give it a question and candidate answers; it returns a label and a probability for each candidate.

• Native vLLM serving on Linux: scheduling, batching, compilation, KV cache, and metrics.
• Apple Silicon preview: run supported text models through MLX or PyTorch MPS, and Valen multimodal decisions through MLX.
• Structured decisions: Choice, Noul (yes/no), and Score (ordered levels) over HTTP.
• Multimodal inference: Valen, vjev, RSI-Jev v4.0-VL, JevAny Pointer, and Clef accept text and images. Valen and Clef also accept short videos; platform availability and request formats are model-specific.
• RadixFlow (experimental, opt-in): bounded admission and state-ready traversal fill shared-prefix execution rounds. See configuration and pressure-test results.
• Automatic setup: start a supported Hugging Face model ID with one command.

Getting Started

On Linux with an NVIDIA GPU, install vLLM Jev with uv from the repository directory:

uv pip install .

For development, use an editable installation.

Start a server with a supported Hugging Face model ID:

vllm-jev serve ZefanCai/Open-Jev-2B

The command downloads and prepares the model, then starts native vLLM at http://127.0.0.1:8795. Later runs reuse the prepared checkpoint.

On an Apple Silicon Mac with macOS 15 or newer, set up once from the repository directory, then serve:

source scripts/install_mac.sh
vllm-jev serve Valen-Team/Valen-Preview-0923

The command downloads and prepares the model on first use. Send text or images to /v1/systemone for Choice, Noul, and Score decisions. For another Mac model, replace the model ID with a supported one from the table below.

GPU, port, and serving options

CUDA_VISIBLE_DEVICES=0 vllm-jev serve ZefanCai/Open-Jev-2B \
  --gpu-memory-utilization 0.90 --port 8795

Add standard vllm serve flags to override the defaults. See serving options for storage locations and other flags.

Visit our documentation to learn more.

• Installation
• Quickstart
• List of Supported Models

Supported Models

Choose a checkpoint for your platform and run its command. Linux uses native vLLM; Apple Silicon uses MLX, or PyTorch MPS for Laya.

The RSI-Jev Linux adapter was contributed by Shanghua Gao (PR #2), and the Clef adapter by Arcobalneo (PR #4). See the RSI-Jev and Clef guides for input limits and request formats.

The JevAny Pointer adapter was contributed by heshien / AsirAsir02 (PR #9). See the JevAny guide for requests and limits.

Additional decision models (experimental)

These text-only checkpoints use the same command and /v1/systemone API for Choice, Noul, and Score. Listed platforms have been tested.

JevK5 currently supports 2–16 options per question. In concurrent Linux batches, Kev-0.8B can change its selected label; This-That and Decider-2B can also change the most likely Score level. These consistency limits remain under investigation.

The optional CUDA setting VLLM_JEV_BF16_MATMUL=no_splitk can reduce batch-dependent numerical drift. Kev label changes can remain, including on A800. See BF16 precision for usage and performance tradeoffs.

See the model guide for limits and request examples.

Example

With the server running, send a Choice request:

curl -sS http://127.0.0.1:8795/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{"state":"I was charged twice and want a refund.","questions":{"intent":{"type":"choice","instructions":"Choose the customer intent.","criteria":{"billing":"A payment or refund issue","technical":"A malfunction or setup issue","other":"Another request"}}}}'

The response contains answers.intent.choice and answers.intent.probabilities for the supplied labels.

See the user guide for supported models, serving options, and Choice, Noul, and Score examples. The plugin targets vLLM 0.29.0 and Python 3.12+.

For image questions, start a supported vision model and follow its image request example. Short MP4 videos are supported by Valen on Linux and Mac, and by Clef on Linux.

RadixFlow pressure comparison

RadixFlow benchmark results

RadixFlow combines bounded admission, state-ready traversal and complete-state copying for shared-prefix scheduling. It is experimental and opt-in on Linux; ordinary serving defaults remain unchanged. Enable it.

Open-Jev-2B/9B throughput below is four-candidate decisions per second, averaged over three fresh-service runs. Native vLLM uses the same checkpoint/plugin, prefix caching and synchronous eager configuration. Short and long inputs come from RACE and QuALITY calibration questions; each condition has 32 requests per run.

Across our tested conditions, RadixFlow delivers up to 53.3% higher decision throughput than native vLLM with prefix caching, measured on Open-Jev-2B with short inputs at concurrency 1 and averaged over three fresh-service runs.

The author comparison includes differences in backend, batching and cache support; its installed runtime uses the published PyTorch fallback without optional FLA/causal-conv1d kernels. These gains are workload-dependent; the short C1 gain includes admission/pooling changes, while long C8/C32 record confirmed dependency bypasses. Full results include variation, latency, worker steps, host time, actual token reuse and output agreement.

A full-attention OpenJev-0.6B control also covers concurrency 1/8/32. It improves low-concurrency throughput but regresses at higher concurrency and changes 15/288 paired selected answers; it is not a recommended RadixFlow deployment case.

Updates

2026-10-14

• Added the inline RadixFlow explainer video and the story behind its cache-to-scheduling investigation. RadixFlow is explicitly experimental and disabled by default.
• Added --radixflow true|false with --radixflow / --no-radixflow aliases, shared by Jev and ordinary generative-model serving. Existing policy flags remain compatible.
• Added reusable Rust-library configuration through VLLM_JEV_RADIXFLOW_LIBRARY. The integration is model-independent within supported native prefix-cache configurations; incompatible cache/scheduler settings are rejected explicitly. Usage.

2026-10-13

• Added opt-in RadixFlow scheduling for synchronous Linux serving with prefix caching. The default serving path is unchanged; setup and supported configurations.
• Added three-run pressure comparisons on Open-Jev-2B and Open-Jev-9B, plus an OpenJev-0.6B compatibility control. Results include worker-round counts, actual token reuse and selected-answer agreement. Results and measurement details.
• Added Linux text/image serving for JevAny Pointer 4B and 27B, contributed by heshien / AsirAsir02 (PR #9).
• Fixed automatic model preparation and explicit protocol selection. JevAny supports all six context_order permutations while preserving its Pointer readouts; native remains the default.
• Verified serving, image handling, prompt ordering and cancellation recovery with regression checks and real-model stress tests.

Earlier updates

2026-10-10

• Added context_order with all six instructions/criteria/state permutations across supported models. Existing presets remain compatible; see the configuration guide.
• Preserved Clef's readout placement and native image/video structure.
• Passed 184 regression checks, 27,300 synthetic requests and an independent 8,000-request retest, including media and cancellation checks. Native order remains the default.

2026-10-09

• Added --decision-template native|instructions-first|path.json. Native prompts remain the default; configuration examples show how to select a preset or load a JSON file.
• Open-Jev-2B/9B can place instructions before the state, or use custom Choice/Score and Noul text. The optional {options} placeholder puts the full fixed candidate list before the changing state while retaining the selected candidate and scoring cue at the end. Thanks to xxyyy's request in #7.
• All model adapters support state_template for text/JSON objects or lists and instruction_template for explicit text instructions. Chat/media envelopes keep their native structure; unsupported full-layout overrides are rejected.
• Templates are loaded at startup and apply to System One and the available Choice/batch routes. Cross-request prefix reuse still requires matching tokens and a shared, trusted cache_salt on supported backends. Restart to change the template; validate custom prompts on your own labelled data because outputs can change.
• Verified 128 regression tests and an independent 2,580-request HTTP stress run, including 80 image requests, concurrency up to 32, cache isolation, and cancellation recovery. Score probabilities and near-tied levels can still vary with batching/cache state.

2026-10-08

• Moved RSI-Jev v3.0/v4.0-VL and Clef/Clef-Flash into the main Supported Models table for Linux.
• Added optional cross-request prefix reuse on /v1/systemone for native Open-Jev, contributed by Shelter Zhou / Sheltercosmo (PR #6). The guide explains trusted salt assignment and limits.
• Rechecked Open-Jev-2B shared-prefix serving with 3,320 requests at up to 32 concurrent clients, covering mixed decision types, cache isolation, and client-disconnect recovery.

2026-10-05

• Added an experimental online prefix cache for repeated Open-Jev-2B text requests on Linux, disabled by default.
• Added bounded cache eviction and state cleanup for cancellation, cache reset, and full-context pooling requests in the experimental path.
• Added an optional CUDA BF16 precision mode to reduce batch-dependent rounding, contributed by Qiao / wocqcm2 (PR #5).
• Kept precision settings fixed for each server process so compiled models use the intended mode. Restart the server to change it.

2026-10-04

• Fixed RSI-Jev requests at the context limit, cached model selection, and invalid media handling.
• Fixed the Linux helper installer and simplified the setup and model guides.
• Added Clef 27B alongside Clef-Flash 9B, with text, image, and video decisions on Linux.
• Added experimental Linux serving for RSI-Jev v3.0/v4.0-VL by Shanghua Gao (PR #2) and Clef-Flash by Arcobalneo (PR #4).
• Added aggregate request limits for Clef and fixed platform and reference-test issues.
• Added optional RSI-Jev shared-state tokenization to reduce preparation time for multi-question requests.

2026-09-29

• Added experimental adapters for Kev, Decider, Mica, This-That, and JevK5, with tested Linux and Mac availability listed above.

2026-09-28

• Added experimental video input for Valen on Linux and Apple Silicon, including variable-frame-rate clips. Usage.

2026-09-27

• Improved Apple Silicon text serving with adaptive batching, prefix reuse, and bounded memory caching.
• Added optional CUDA Graph readouts for Laya multilingual on Linux.
• Added optional repeated-image reuse for Valen on Mac. See serving options.

2026-09-26

• Added all three Laya checkpoints on Linux and Apple Silicon: English, multilingual, and typed decisions.
• Added Tiny-Jev serving on Apple Silicon with its marker readout.
• Added Open-Jev-2B serving on Apple Silicon with its scalar decision head.
• Added local text and image decisions on Apple Silicon with OpenJev-0.6B and Valen.
• Set up with one script, then start a model with the same vllm-jev serve command.
• Kept Mac server health checks responsive during inference.

2026-09-25

• Added multimodal decision inference with support for text and images.
• Added a fused Tiny-Jev head: 12.1% higher throughput in the measured 16-option, 16-concurrent workload.
• On 500 Valen image questions, median latency fell from 231.9 to 77.3 ms (3.0×) with 86.8% target-support accuracy on both paths. Results.

Inference performance

Inference benchmarks and measurement notes

Apple Silicon

Measured on an Apple M5 with 16 GB unified memory, using MLX: 96 text Choice requests for OpenJev and 42 GameQA Maze image questions for Valen. Requests are sequential after warmup; model loading is excluded.

Valen selected the same answers as the Linux reference on all 42 questions; both answered 17 correctly. Probability values differ between backends. Mac executes one model request at a time; concurrent requests queue.

NVIDIA GPUs

Each row uses the same model, input, and A800 GPU for both paths. Lower latency and higher throughput are better. Measurement methods are noted below.

Paired values: Without vLLM Jev → With vLLM Jev.

Benchmark notes

• ¹ OpenJev-0.6B uses the author's CUDA scorer as its baseline.
• ² Tiny-Jev uses a serialized HTTP wrapper around the author's Python API; gains include batching.
• ³ Valen uses sequential offline inference on 500 image questions. Both paths reached 86.8% target-support accuracy; choices agreed on 98.8%. Probability values differed, with a largest per-question difference of 0.295. HTTP time is excluded. A separate 500-request HTTP run at concurrency 8 reached 40.6 req/s (193.4 ms median latency).
• ⁴ vjev uses the author's Python scorer versus vLLM Jev HTTP on 100 identical game images and the same A800. Only the vLLM side includes HTTP time. This measures serving speed, not game accuracy.
• ⁵ Laya compares the published laya-serve 0.3.20 with vLLM Jev HTTP on the same A800 and 500 fixed four-choice questions. Labels differed on 6/500 questions; the largest option-probability difference was 0.0222. Laya's direct predict_batch SDK can be faster for many states sharing one question schema; the HTTP figures do not measure that bulk path. The optional TileLang fast path was unavailable with this server's CUDA toolkit.
• ⁶ The two additional Laya checkpoints use the same 500 English MMLU questions and A800 HTTP setup as ⁵. Their labels differed from the author's runtime on 7/500 (multilingual) and 2/500 (typed) questions. This tests serving on a shared English workload, not multilingual or typed-task quality.

Contributing

Report bugs and feature requests in Issues. Use Discussions for questions and ideas. Submit code changes as pull requests; the maintainer reviews them before merging.

License

Apache-2.0. The Choice prompt follows MIT-licensed Open-Jev; see third-party notices.

Citation

If you use vLLM Jev in your research, please cite our paper: RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving.

@misc{jing2026radixflow,
  title         = {{RadixFlow}: Filling Execution Bubbles in Prefix-Cached Hybrid {LLM} Serving},
  author        = {Huihao Jing and Wenbin Hu and Shaojin Chen and Haochen Shi and Haoyu Huang and Sirui Zhang and Yuxuan Liu and Zhongwei Xie and Haoran Li},
  year          = {2026},
  eprint        = {2610.18143},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2610.18143}
}