chatllm.cpp

foldl/chatllm.cpp is ranked 12,557 among these 8 peers; codefuse-ai/ModelCache is ranked 12,556.

Deliver accurate or better results than other implementations o-c-c-a-s-i-onally. | Supported Models | Download Quantized Models | **What's New:** 2026-10-02: Intern-Decision 2026-09-27: Spark-2. 5 2026-09-25: TeleOCR 2026-09-08: MiniCPM5-2B 2026-08-22: Instella-MoE 2026-08-04: Nanbeige4. 2 2026-07-24: G9v3 2026-07-16: Introspection with lens 2026-07-15: OvisOCR2 2026-06-27: PaddleOCR-VL 2026-06-22: Mellum-2 2026-06-14: HY

-MT2 2026-05-28: MiniCPM5 2026-05-05: Gemma-4 2026-03-28: InternVL3. 5 2026-03-27: Qianfan-OCR 2026-03-22: Penguin-VL 2026-03-06: Qwen3. 5 2026-03-03: GLM-OCR 2025-10-10: I can draw: Janus-Pro 2025-06-21: I can hear: Qwen2-Audio 2025-05-23: I can see: Fuyu 2025-05-21: Re-quantization when loading (e. ) 2025-05-17: I can speak: Orpheus-TTS 2025-03-24: GGMM file format

2025-02-21: Distributed inference 2025-02-10: GPU acceleration 🔥 2024-12-09: Reversed role 2024-11-21: Continued generation 2024-11-01: generation steering 2024-06-15: Tool calling 2024-05-29: ggml is forked instead of submodule 2024-05-14: OpenAI API, CodeGemma Base & Instruct supported 2024-05-08: Layer shuffling Features [x] Accelerated memory-efficient CPU/GPU inference with int4/int8 quantization, optimized KV cache and parallel computing; [x] Use OOP to address the similarities between different

_Transformer_ based models; [x] Streaming generation with typewriter effect; [x] Continuous chatting (content length is virtually unlimited) [x] Retrieval Augmented Generation (RAG) 🔥 [x] LoRA; [x] Python/JavaScript/C/Nim Bindings, web demo, and more possibilities. cpp repository into your local machine: If you forgot the flag when cloning the repository, run the following command in the folder: Quantize Model **Some quantized models can be downloaded on demand. ** Install dependencies of : Use to transform models into quantized GGML format. For example

, to convert the _fp16_ base model to q8_0 (quantized int8) GGML model, run: Use to specify model's name in English. Optionally, use to specify model's name in another language. Use to specify the path of the LoRA model to be merged, such as: Note: Appropriately, only HF format is supported (with a few exceptions); Format of the generated files is different from the one (GGUF) used by . Customized Quantization It's possible to specify quantization for each tensor using , where

is a regex pattern to match against tensor names. In this example, all tensors are using quantization except embeddings are using . Example: Vulkan acceleration together with RPC and backend dynamic loading: Run Now you may chat with a quantized model by running: To run the model in interactive mode, add the flag. For example: In interactive mode, your chat history will serve as the context for the next-round conversation. Plan Nim All Python scripts are going to be rewritten

in Nim, with following exceptions: when is used Acknowledgements This project is started as refactoring of ChatGLM. cpp, without which