TensorSharp

English | 中文

TensorSharp is a .NET 10 inference engine for local GGUF models. Run it on Windows, macOS or Linux through the CLI, browser chat, or Ollama/OpenAI-compatible APIs, or embed it in your own .NET application. Use managed C# CPU kernels or native CUDA, Metal and Vulkan backends, with support that varies by model.

Current source covers text and reasoning, multimodal input, embeddings, image generation/editing, video with audio, personal knowledge collections, and Agent Skills with code tools. TensorSharp also powers TensorAgent, the local app for iPhone, iPad, Mac and Windows. Start below, or check the project status for capabilities and validation limits; knowledge and response steering were added after the v2026.10.10 tag and require a build containing those changes.

Highlights

• Local models, several interfaces. One engine for CLI use, browser chat and Ollama/OpenAI-compatible APIs, across managed CPU and native accelerator backends.
• Text and multimodal models. Dense and MoE GGUF models, reasoning, image/audio input and document questions. See Supported models for each family's capabilities.
• Embeddings and media generation. Text/code embeddings, Qwen-Image-2.1 and its 8-step Turbo variant generation and masked editing, and video models including MiniMax-H3 with audio.
• Efficient serving. Continuous batching and a paged, Radix prefix-shared KV cache are on by default. Speculative decoding and multi-GPU placement are available for supported models. See Features.
• Agent Skills and code tools. TensorSharp.AgentHost adds file, shell and document workflows. Server and TensorAgent chats also support bounded sub-agent delegation with private workspaces and read-only defaults.
• Personal knowledge collections. Upload files or nested folders, and import saved chats with their attachments and artifacts. Select collections per chat, retrieve cited excerpts with page/row/time locations, preview originals, or let code agents open full datasets. Local BM25 works by default; optional GGUF embeddings add semantic search, and compatible media models index images, audio and sampled video. Extraction status and warnings show coverage limits.
• Steer an active response. Submit a new instruction while the CLI, server Web UI or TensorAgent is reasoning or answering. The next response uses the conversation, partial output and your guidance; Stop remains available.
• Recorded comparisons. Benchmarks against llama.cpp use identical GGUF files and hardware; results apply to the measured model, backend and workload. See Benchmarks.
• TensorAgent apps. Local chat, attachments, skills and saved work on phones and desktops. See the Mac/Windows installation guide and app coverage notes.

Quick Start

TensorSharp CLI and server

The Releases page provides self-contained CLI and Server archives for Windows x64 (CPU/CUDA), Linux x64 (CPU/CUDA), and macOS arm64.

To build from source you need the full .NET 10 SDK (how to install it), git, curl, CMake 3.20+, and the toolchain for your GPU. Then run the verified Gemma 4 E4B model (7.48 GiB). On Windows with an NVIDIA GPU (PowerShell):

git clone https://github.com/zhongkaifu/TensorSharp.git; Set-Location TensorSharp
New-Item -ItemType Directory -Force models | Out-Null
curl.exe -L --fail "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models\gemma-4-E4B-it-Q8_0.gguf
'Answer in one short sentence: what is TensorSharp?' | Set-Content prompt.txt
$env:TENSORSHARP_GGML_NATIVE_ENABLE_CUDA = 'ON'
dotnet run --project TensorSharp.Cli -c Release -p:TensorSharpSkipMlxNative=true -- --model models\gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --max-tokens 128 --backend ggml_cuda

On other machines, change the backend (see Pick a backend):

• macOS (Apple Silicon): drop the CUDA environment variable and use --backend ggml_metal.
• Linux + NVIDIA: prefix the dotnet run with TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON and use --backend ggml_cuda.
• AMD / Intel / NVIDIA Vulkan: set TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON and use --backend ggml_vulkan.

Host the same model as a server: a browser chat at plus Ollama- and OpenAI-compatible APIs.

dotnet run --project TensorSharp.Server.Host -c Release -p:TensorSharpSkipMlxNative=true -- --model models/gemma-4-E4B-it-Q8_0.gguf --backend ggml_cuda --max-tokens 512

The server listens on 0.0.0.0:5000 with no built-in authentication or TLS; keep it behind a firewall or an authenticated HTTPS reverse proxy.

Pick a backend

The Getting started guide has the rest: installing the SDK on each platform, multi-GPU and multi-node runs, NVIDIA DGX Spark, multimodal input, embeddings, and making it fast. Every option is in the CLI and Server references, and both programs print them with --help.

dotnet build TensorSharp.slnx also builds TensorAgent's available desktop heads and the iOS simulator head on Apple Silicon when the selected SDK has the required MAUI workloads and staged native/Python files. Missing prerequisites skip the affected app head with a warning; see TensorAgent build instructions.

TensorAgent Desktop: download, install, chat

Download the latest release and expand Assets. Choose a file beginning with tensoragent-desktop-:

The app includes its .NET runtime and native engine. Windows needs WebView2 Evergreen Runtime if missing; Python/Node are optional tools for skills. The current built-in catalog needs at least 12 GB system RAM, and model weights download separately. Open ☰ → Models → Download → Use, then type a message. The Desktop user guide covers package verification, unsigned-app prompts, first-run setup, attachments, skills, updates and troubleshooting. Historical releases may have no Desktop assets until a release runs the updated workflow; iPhone/iPad remain source builds.

Supported model families at a glance

• Text, reasoning, and multimodal LLMs: DeepSeek V4 Flash / V4.1 Flash, GLM 5.x, Gemma 4, Qwen 3.5 / 3.6 / 3.8 27B, Qwen 3.8 Flash Next, Bonsai2 (Qwen family), GPT OSS, Nemotron-H, Mistral 3, Hunyuan Dense, and Muse-Glimmer.
• Text diffusion: DiffusionGemma, including Jev typed decision inference at /v1/systemone, over text, images, uploaded documents, sampled video frames and audio transcripts (configured ASR companion).
• Image generation/editing and video generation: Qwen-Image-2.1, MiniMax-H3 (video + stereo audio), and Wan 2.1 / 2.2.
• Text and code embeddings: BERT / XLM-R encoders — Snowflake Arctic Embed L v2.0 and all-MiniLM-L6-v2.

Backend, modality, feature support, and validation coverage vary by model. See the supported models tables, the model cards, and the embedding guide for details.

TensorAgent's current catalog has 38 entries, grouped into searchable model families with memory and download filters, including Qwen-Image-2.1 Turbo at 4-bit and 8-bit quantizations and fourteen image LoRA plug-ins. The expanded catalog includes experimental entries whose memory tiers and app/device behavior have not all been measured. Qwen3.8 Flash Next uses SSD-backed weights; its Q2 package was measured on a 48 GB Mac, and IQ1_M has separate Windows catalog-host validation. Multi-GPU --layer-split and --tp are separate controls; support and performance depend on the architecture and quantization. See TensorAgent's coverage notes and the model cards. Desktop installers appear in releases built with the updated Release Binaries workflow, while older releases may lack them.

See it in action

One engine, four ways to use it, each an unedited capture of a real run.

TensorSharp.Cli
Models in your terminal

TensorSharp.Server.Host
Web UI chat and Ollama/OpenAI-compatible APIs

TensorAgent · iPhone simulator
CPU inference and in-app Python; this is a simulator capture

TensorAgent on Mac
A saved Qwen-Image 2.1 edit in the Mac app

What each run shows, step by step: Screenshots.

Benchmarks

TensorSharp and llama.cpp run identical GGUF files on the same NVIDIA RTX 3080 Laptop GPU (16 GB), each on its GGML CUDA and Vulkan builds. Each number is TensorSharp's speedup over llama.cpp on the same backend (geomean, single-stream, greedy, MTP off); above 1.0× means TensorSharp is faster.

What these numbers mean, how to rerun them, and the head-to-heads of models too large for this GPU: Benchmarks.

Documentation

New here? The sections above are all you need to get running. Everything else is detailed reference:

Current Status

Actively developed, and the source tree runs ahead of the published packages.

Per-area detail (which architecture runs on which backend, which features each family supports, and the known limits) is in the status matrix.

Author

Zhongkai Fu

License

See LICENSE for details.

Learn with the books

Qwen inference and agentic runtimes	Gemma 4 and multimodal inference
Building LLM Inference Engines and Agentic Runtimes from Scratch: Qwen Dense and MoE Models with TensorSharp and TensorAgent	From Tensors to Tokens: Building a Multimodal LLM Inference Engine from Scratch with TensorSharp and Gemma 4 E4B
Build Qwen dense/MoE inference and controlled agent workflows in C#. Follow tensors, tokenization, attention, expert routing, quantization, and caching through GPU acceleration, multimodal execution, tools, skills, sandboxed code, and desktop/mobile deployment with TensorSharp and TensorAgent.	Build a multimodal inference engine in C#/.NET with Gemma 4 E4B, from tensors, GGUF model loading, quantization, and tokenization to text, image, video, and audio execution. Connect correctness checks and serving optimizations to the running TensorSharp code.
Buy on Amazon	Buy on Amazon

Explore both books and their repository reading paths

TensorSharp/TensorAgent is free. If you like it, a coffee keeps the work on it going.