turboderp

A fast inference library for running LLMs locally on modern consumer-class GPUs

A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs