ModelCache

codefuse-ai/ModelCache is ranked 12,556 among these 8 peers; alibaba/BladeDISC is ranked 12,555.

09] Added Redis Search to store and retrieve embeddings in multi-tenant. This can reduce the interaction time between Cache and vector databases to 10ms. 10] Integrated LLM embedding frameworks such as 'llmEmb', 'ONNX', 'PaddleNLP', 'FastText', and the image embedding framework 'timm' to bolster embedding functionality. 20] Integrated local storage, such as sqlite and faiss. This enables you to initiate quick and convenient tests. Introduction Codefuse-ModelCache is a semantic cache for large language models (LLMs). By caching pre-generated

model results, it reduces response time for similar requests and improves user experience. This project aims to optimize services by introducing a caching mechanism. It helps businesses and research institutions reduce the cost of inference deployment, improve model performance and efficiency, and provide scalable services for large models. Through open-source, we aim to share and exchange technologies related to large model semantic cache. Architecture Quick start You can find the start script in and . : A quick test service that

embeds SQLite and FAISS. : The standard service that requires MySQL and Milvus configuration. 8 or above Package installation Start service Start demo 1. Download the embedding model bin file from Hugging Face. Start the backend service: Service Startup With Docker-compose 1. Download the embedding model bin file from Hugging Face. Configure docker network, only need to execute once 3. Execute the docker-compose command Start normal service Before you start standard service, do these steps: 1. Install MySQL and

import the SQL file from . Download the embedding model bin file from Hugging Face. Start the backend service: Visit the service The service provides three core RESTful API functionalities: Cache-Writing, Cache-Querying, and Cache-Clearing. Write cache Query cache Clear cache Function comparison We've implemented several key updates to our repository. We've resolved network issues with Hugging Face and improved inference speed by introducing local embedding capabilities. Due to limitations in SqlAlchemy, we've redesigned our relational database interaction module

for more flexible operations. We've added multi-tenancy support to ModelCache, recognizing the need for multiple users and models in LLM products. Lastly, we've made initial adjustments for better compatibility with system commands and multi-turn dialogues. Features In ModelCache, we incorporated the core principles of GPTCache. ModelCache has four modules: adapter, embedding, similarity, and data_manager. The adapter module orchestrates the business logic for various tasks, integrate the embedding, similarity, and data_manager modules. The embedding module converts text

into semantic vector representations, and transforms user queries into vectors. The rank module ranks and evaluate the similarity of recalled vectors. The data_manager module manages the databases. To make ModelCache more suitable for industrial use, we made several improvements to its architecture and functionality: [x] Architectural adjustment (lightweight integration): Embedded into LLM products using a Redis-like caching mode Provided semantic caching without interfering with LLM calls, security audits, and other functions Compatible with all LLM services [x] Multiple

model loading: Supported local embedding model loading, and resolved Hugging Face network connectivity issues Supported loading embedding layers from various