surogate

Surogate Engine

Native C++/CUDA LLM training and serving: GGUF, NVFP4, BF16, FP8.

From your first fine-tune to hundreds of concurrent requests.

Website · Documentation · Benchmarks · Quickstart · Models · Examples

136,200 tok/s
Training · 4× RTX 5090
Qwen3-0.6B · FP4 LoRA

2.53× training throughput
vs. Unsloth · 1× H100
Qwen3-0.6B · BF16 on both

802 tok/s
Serving · one user · 1× RTX 5090
Qwen3.5-0.8B · native GGUF

7.0× serving throughput
vs. llama.cpp · 8× RTX 5090
GLM-5.3-Flash · 16 users · decode

Selected results from the repository's training benchmarks and serving benchmarks. Workloads and comparison details below.

Surogate puts training and serving in one toolkit, with dedicated native engines built to push NVIDIA hardware. Pretrain a model, fine-tune with LoRA or QLoRA, optimize with GRPO or DPO, distill a teacher, and serve the result through familiar HTTP APIs.

Speed drives the design: compiled training graphs, fused CUDA kernels, low-precision tensor cores, concurrent serving, and deliberate control over every byte of GPU memory. Scale from a single workstation to multiple GPUs and training clusters, or use system RAM to work with models larger than your cards.

Speed you can measure

Serving: fast for one user. Fast under load.

Native GGUF decoding at 802 tokens/s on one RTX 5090. A 200 GB MoE serving 16 users at 7× llama.cpp's aggregate decode throughput on eight cards.

Responsiveness matters, too. At 100 users, Qwen3.5-4B delivers 40 ms median time to first token, versus 230 ms for vLLM. At 16 users, GLM-5.3-Flash delivers 1.59 seconds, versus 55.13 seconds for llama.cpp.

Training: more experiments per GPU-hour

53,900 training tokens/s on a single H100. 136,200 on four RTX 5090s.

Need more throughput? Qwen3-0.6B FP4 LoRA scales from 36,400 tok/s on one RTX 5090 to 136,200 tok/s on four: 3.74× aggregate throughput. Qwen3-8B reaches 27,100 tok/s on the same four-card setup.

Every model and GPU we have measured

Each comparison ran both engines on the same kind of GPU with the same requests. A ratio above 1× favours Surogate. The rows where the other engine leads stay in the table.

Serving. Decode and prefill are tokens/s summed over all users, except where a row says "per user".

Also measured, without another engine alongside: single-user decode of Qwen3-0.6B at 685 tok/s and Qwen3.5-0.8B at 563 tok/s on an H100; Qwen3-0.6B at 220 tok/s, Qwen3.5-4B NVFP4 at 73 tok/s, Gemma 3 1B Q8_0 at 138 tok/s and Qwen3.6-27B NVFP4 at 12.0 tok/s on a DGX Spark, the last at the Spark's memory-bandwidth limit.

Training. LoRA, tokens/s.

RTX 5090 serving rows: 2026-08-30 to 2026-09-07, cards at a 400 W limit, vLLM 0.27.1 and llama.cpp CUDA builds. H100 and DGX Spark rows: 2026-10-05 to 2026-10-09, vLLM 0.31. On the DGX Spark, vLLM's FP8 KV cache fails to start, so vLLM ran a BF16 cache; Surogate ran a BF16 cache for the 35B's short-prompt rows and an FP8 cache for its long-context and MTP rows, for Flash-Next and for Qwen3.8-27B. Qwen3.8-Flash-Next's NVFP4 release needs 123.5 GiB in vLLM on a 121.7 GiB Spark; a public vLLM preview with MTP is reported at 31-41 tok/s for one user there (not measured here). Surogate's Flash-Next rows run the 105 GB artifact converted from that release, and its prefill row is against llama.cpp on Unsloth's UD-IQ4_XS GGUF. Benchmark prose is the repeated 512-token text the throughput rows use, where up to 95% of MTP drafts are accepted; "real text" means distinct Wikipedia passages, where 52-54% are. Full methods: serving benchmarks, training benchmarks, and the pull requests behind each result (#296-#309).

Two engines. One workflow.

Training engine

From raw text to specialized models, with Python configuration and native C++/CUDA execution.

Explore training modes, precision recipes, memory management, GRPO, DPO, and distillation.

Serving engine

A native C++/CUDA HTTP server built for quick responses, concurrent workloads, and efficient model placement.

Formats, LoRA, vision, speculation, and placement options depend on the model family. Multi-model hosting, sleep mode, and DFlash currently require one GPU. Serving guide → · CLI reference → · Deployment examples →

Supported models

Training and serving have different architecture coverage. Model dimensions come from the checkpoint; the examples below are representative sizes, with precision and memory requirements depending on the configuration.

Training

Additional architecture definitions include Gemma 3, LFM2-MoE, and LFM2-VL. Check their implementation and validation coverage before using a new checkpoint. DeepSeek-V4, Flash-Next, and GLM-5.3-Flash training definitions still have deferred components and are not listed as ready-to-train models.

Serving

For per-family format support, conversion behavior, and hardware details, see the serving model guide.

Quickstart

Install

Use Linux x86_64 and Python 3.12 with a supported NVIDIA GPU and CUDA 13 (driver 580 or newer), or a DGX Spark (GB10, Linux aarch64). See hardware for the separate training and serving GPU targets.

curl -LsSf https://github.com/invergent-ai/surogate/releases/latest/download/install.sh | bash
source .venv/bin/activate

The installer verifies the install and downloads the example configurations. The wheel is also installable on its own, with no index flags — every dependency resolves from PyPI:

pip install https://github.com/invergent-ai/surogate/releases/latest/download/surogate-+cu130-cp312-abi3-manylinux_2_39_x86_64.whl

On a DGX Spark the installer picks the manylinux_2_39_aarch64 wheel, built for GB10 only.

There is no CUDA 12 package. The serving engine's W8 and NVFP4 kernels declare more shared memory per block than a CUDA 12 toolkit will assemble for the RTX line, so CUDA 13.0 is the floor for building it, and a package that serves is the product. The engine carries its own FFmpeg, numa and ICU; it expects the system's glib and X11 client libraries, which a minimal server image may lack (Ubuntu: libglib2.0-0t64 libx11-6 libxext6 libxrender1). The installer checks.

Serve a model

surogate serve Qwen/Qwen3.5-0.8B \
  --served-model-name surogate \
  --port 8080 --max-model-len 4096 \
  --max-num-seqs 16 --kv-capacity auto

The first launch prepares and caches the checkpoint. Later launches reuse that preparation. You can also pass a local safetensors directory or a supported .gguf file.

Send a streaming request from another terminal:

curl -N http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "surogate",
    "messages": [{"role": "user", "content": "Explain why low latency matters."}],
    "max_tokens": 256,
    "stream": true
  }'

Point existing OpenAI-compatible clients at http://127.0.0.1:8080/v1. To try a Blackwell NVFP4 checkpoint, use a supported export such as nvidia/Qwen3.6-27B-NVFP4. More serving examples →

Train an adapter

Save this as train.yaml:

model: Qwen/Qwen3-0.6B
output_dir: ./output

recipe: bf16                  # fp8-hybrid for FP8; nvfp4 for Blackwell FP4
lora: true
lora_rank: 16
lora_alpha: 32
lora_target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]

sequence_len: 2048
sample_packing: true
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
learning_rate: 2e-4
max_steps: 100
save_steps: 50

datasets:
  - path: mlabonne/FineTome-100k
    type: auto

surogate sft train.yaml

Choose a precision recipe for your GPU, or start with the checked-in training examples. For memory-constrained runs, see QLoRA and CPU offloading.

Serve what you trained

The completed training run exports its final adapter directly to output/. Merge it into its base model, then start the server:

surogate merge \
  --base-model Qwen/Qwen3-0.6B \
  --checkpoint-dir output \
  --output merged

surogate serve merged --served-model-name my-finetune --port 8080

Use "model": "my-finetune" in requests. To create a quantized GGUF for serving:

surogate quantize --model merged --output merged-Q4_K_M.gguf --type q4_k_m
surogate serve merged-Q4_K_M.gguf --served-model-name my-finetune

Run one server at a time on the same port. Compatible adapters can also be loaded at runtime with --enable-lora.

Run GRPO on one GPU

Load the base model once for both serving and training. Native co-locate mode supports BF16 safetensors models with LoRA for text rollouts across the supported training families, excluding Nemotron. Dense Qwen3 and Qwen3.5 use optimized serving; other families reuse the training model for generation and recompute the prefix per token, which is slower. MoE models require lora_dtype: bf16, and the entire base must fit on one GPU. It generates a batch of rollouts, pauses generation for the training update, then generates the next batch with the updated adapter. Per-step adapter updates stay in GPU memory.

Create the three configuration files from the Single-GPU GRPO guide, using backend: surogate for inference and a fresh output directory:

CUDA_VISIBLE_DEVICES=0 surogate grpo-colocate \
  --train train.yaml --infer infer.yaml --orch orch.yaml

Training buffers remain reserved during generation, so choose a model and context length that fit your GPU. Quantized bases, multiple GPUs, and checkpoint resume are not yet supported in this native mode. See the GRPO guide for the separate-GPU runner.

Docker and source builds

The container is ghcr.io/invergent-ai/surogate:latest (also tagged latest-cu130 and by release version).

# From the directory containing train.yaml; output stays in ./output on the host.
docker run --gpus all --rm \
  -v "$PWD:/workspace" -w /workspace \
  ghcr.io/invergent-ai/surogate:latest-cu130 sft train.yaml

For development, clone the repository and install with a CUDA toolkit, NCCL development libraries, and FFmpeg/libcurl development packages for serving. See CMake for the full build configuration:

git clone https://github.com/invergent-ai/surogate.git
cd surogate
uv venv --python 3.12
source .venv/bin/activate
uv pip install -e .

make serve-build rebuilds the serving engine in a source checkout. Installation details →

Why it is fast

• Compiled training graphs. A Python DSL defines the model; ahead-of-time autodiff produces its backward graph so the native runtime can plan execution and memory across both passes.
• Kernels that do more per launch. Fused operations, specialized matrix multiplication, low-precision tensor cores, and CUDA graphs reduce dispatch overhead and memory traffic.
• Serving that keeps work moving. Continuous batching, chunked prefill, prefix reuse, and speculative decoding improve throughput and latency for different request patterns.
• Memory treated as part of the engine. Planned buffer reuse, asynchronous offload, elastic KV caches, and expert placement make more of the available GPU and system memory useful.

Read how training works, the DSL guide, or the serving benchmark analysis.

Hardware

The training and serving engines have separate CUDA build targets. Check the build configuration when targeting a GPU beyond the default serving build.

Explore and contribute

Contributions are welcome: model support, kernels, precision recipes, benchmarks, documentation, and bug fixes. Include the problem, how to reproduce or validate the change, and any GPU or architecture assumptions in your pull request. Open an issue or send a PR.

Apache 2.0 · License · Built by Invergent