glake

antgroup/glake is ranked 18,283 among these 8 peers; langfuse/langfuse-python is ranked 18,319.

07] We release GlakeServe, our LLM serving and KV Cache management using VMM technique. 05] GLake overview and recent update is presented on AICon 2024 (in Beijing, China, 2024-05-17) here [2024. 05] The presentation slides in ASPLOS'24 can be found here [2024. 05] Preview: **serverless** with on-demand GPU memory shink (ckpt to CPU while), keeping process alive and auto backfill upon request coming [2024. 04] Preview: manage **LLM KV cache**, different from PagedAttention, it's flexible and

easy to use. 04] Preview: **cross-stream** GPU memory optimization dev done, will be released soon; integration with PyTorch expand segment is in-progress by AntGroup and Alibaba contributor [2024. 10] Our paper: GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitching, just accepted by **ASPLOS'24** Blog Introduction AI large model training and inference are increasingly confronted with the challenges of memory wall and IO transmission wall, that is, the growth of

GPU memory capacity and IO bandwidth cannot keep up with the growth rate of AI model size. To address these challenges, GLake is an acceleration library and relevant utilites that work at the bottom layer (GPU virtual and physical memory management) and system layer (multi-GPU, multi-path, and multi-tasking) to optimize GPU memory and IO. GLake enables AI training, inference (including converting large models to TensorRT or ONNX Runtime on NVIDIA A10/3090) and DevOps (like Notebook) to

fully utilize the underlying hardware resources, improving training throughput by up to 4 times, saving inference memory by up to 3 times, and accelerating IO transmission by 3~12 times. To use GLake, the simplest way is to replace the underlying library (e. so), though more graceful way is to follow the detailed steps. Motivation **GPU memory bottleneck** GPUs are known for their high computing power and high concurrency. As a peripheral, however, its memory capacity (currently 80GB for mainstream

training GPUs A100 and 24GB for mainstream inference GPUs A10) still restricts the use of its computing power. Especially recently, the growing demand for GPU memory capacity for large models has been much higher than the hardware development of GPU memory. **IO transmission bottleneck** Comparing the GPU computing power and CPU-GPU IO bandwidth of various GPU generations, it is not difficult to find that the limitations of transmission wall are intensifying and will be unlikely to be solved in

the short term. It is worth noting that based on the customized interconnection NVLink, the GPU-to-GPU bandwidth is significantly faster than the PCIe bandwidth. In addition, GPU memory bandwidth (HBM, GDDR) is a performance bottleneck of large model inference. Architecture GLake is designed with a layered architecture. Currently tests and verfications focus on PyTorch and NVIDIA GPUs, we're working on more devices support: **Hardware interface** includes GPUs and interconnection, currently mainly based on NV GPU (supporting NVLink, P2P

, Unified Addressing, VMM, IPC, etc. The interface is adapting to support domestic AI cards, and will consider supporting new interconnections