Langtrace 🔍 is an open-source, Open Telemetry based end-to-end observability tool for LLM applications, providing real-time tracing, evaluations and metrics for popular LLMs, LLM frameworks, vectorDBs and more.. Integrate using Typescript, Python. 🚀💻📊
ezkl is an engine for doing inference for deep learning models and other computational graphs in a zk-snark (ZKML). Use it from Python, Javascript, or the command line.
🔒 Enterprise-grade API gateway that helps you monitor and impose cost or rate limits per API key. Get fine-grained access control and monitoring per user, application, or environment. Supports OpenAI, Azure OpenAI, Anthropic, vLLM, and open-source LLMs.
[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
A Python module for getting the GPU status from NVIDA GPUs using nvidia-smi programmically in Python
关键参数与规格
关键参数与规格
参数
值
语言
C++
类别
Infrastructure
子类别
Inference service optimization
星标数
1,229
分叉数
274
贡献者
2
创建日期
2018-04-27
更新日期
2026-10-08
1 天星标数
0
7 天星标数
0
排名
11,053
上次排名
11,008
排名变化
-45
7 天星标排名
12,870
上次 7 天星标排名
10,085
7 天星标排名变化
-2,785
总星标排名
9,506
上次总星标排名
9,501
总星标排名变化
-5
国家
China
README 摘录
Introduction FeatherCNN is a high-performance lightweight CNN inference library, developed by Tencent AI Platform Department. FeatureCNN origins from our game AI project for King of Glory (Chinese: 王者荣耀), in which we aim to build a neural model for MOBA game AI and run it on mobile devices. FeatherCNN currently targets at ARM CPUs. We will extend it to cover other architecutures in the near future. Comparing with other libraries, FeatherCNN has the following features: **High Performance
** FeatherCNN delivers state-of-the-art inference computing performance on a wide range of devices, including mobile phones (iOS/Android), embedded devices (Linux) as well as ARM-based servers (Linux). **Easy Deployment** FeatherCNN packs everything in a single code base to get rid of third-party dependencies. Hence, it facilitates deployment on mobile platforms. **Featherweight** The compiled FeatherCNN library is small-sized (hundreds of KBs). Please kindly open an issue in this repo for bug reports and enhancement suggests. We
are grateful to user responses and will actively polish this library. Citation FeatherCNN: Fast Inference Computation with TensorGEMM on ARM Architectures (TPDS September 2019, In press, DOI:10. 2939785) Clone hints The FeatherCNN repository has a heavy development history, please only clone the master branch as follows: Detailed Instructions for iOS/Android/Linux **Build From Source** **iOS Guide** **Android Guide** **Android ADB Guide** Usage Model Format Conversion FeatherCNN accepts Caffemodels. caffemodel) into a single binary model (. The convert tool requires
protobuf, but you don't need them for the library. Runtime Interfaces The basic user interfaces are listed in feather/net. Currently we are using raw pointers to reference data. We may provide more convenient interfaces in the near future. Before inference, FeatherCNN requires two steps to initialize the network. The net can also be initialized with raw buffers and FILE pointers. We can perform forward computation with raw buffer consequently. The output can be extracted from the net by the
name of blobs. The blob names are kept consistent with caffe prototxt. BTW, you can also get the blob's data size by calling Performance Benchmarks We have tested FeatherCNN on a bunch of devices, see **this page** for details.
FeatherCNN is a high-performance lightweight CNN inference library, developed by Tencent AI Platform Department.
FeatureCNN origins from our game AI project for King of Glory (Chinese: 王者荣耀), in which we aim to build a neural model for MOBA game AI and run it on mobile devices.
FeatherCNN currently targets at ARM CPUs.
We will extend it to cover other architecutures in the near future.
Comparing with other libraries, FeatherCNN has the following features:
High Performance FeatherCNN delivers state-of-the-art inference computing performance on a wide range of devices, including mobile phones (iOS/Android), embedded devices (Linux) as well as ARM-based servers (Linux).
Easy Deployment FeatherCNN packs everything in a single code base to get rid of third-party dependencies. Hence, it facilitates deployment on mobile platforms.
Featherweight The compiled FeatherCNN library is small-sized (hundreds of KBs).
Please kindly open an issue in this repo for bug reports and enhancement suggests. We are grateful to user responses and will actively polish this library.
Citation
FeatherCNN: Fast Inference Computation with TensorGEMM on ARM Architectures (TPDS September 2019, In press, DOI:10.1109/TPDS.2019.2939785)
Clone hints
The FeatherCNN repository has a heavy development history, please only clone the master branch as follows:
FeatherCNN accepts Caffemodels. It merges the structure file (.prototxt) and the weight file (.caffemodel) into a single binary model (.feathermodel). The convert tool requires protobuf, but you don't need them for the library.
The basic user interfaces are listed in feather/net.h. Currently we are using raw pointers to reference data.
We may provide more convenient interfaces in the near future.
Before inference, FeatherCNN requires two steps to initialize the network.
