AI Rank
EN
← 开源
kennethleungty

Llama-2-Open-Source-LLM-CPU-Inference

Running Llama 2 and other Open-Source LLMs on CPU Inference Locally for Document Q&A

TutorialsAIE roadmapPython
在 GitHub 打开
增长势头
+024 小时新增 Star0.0%
971
Star
205
Fork
+-1
本周
4
贡献者
创建于 2023-07-06 · 更新于 2026-10-14 · 今日第 12311 名
主要开发者
kennethleungtyeltociearmsfasha

kennethleungty/Llama-2-Open-Source-LLM-CPU-Inference 排名第 12,311，有 971 个星标。

同类别比较

kennethleungty/Llama-2-Open-Source-LLM-CPU-Inference 排名第 12,311，此处有 7 个同类条目；rasbt/MachineLearning-QandAI-book 排名第 12,300。

同类别比较
名称排名指标
Azure-Samples/contoso-real-estate12,289
语言:
JavaScript
星标数:
974
分叉数:
529
简介:
Intelligent enterprise-grade reference architecture for JavaScript, featuring OpenAI integration, Azure Developer CLI template and Playwright tests.
aialgorithm/Blog12,292
语言:
Jupyter Notebook
星标数:
974
分叉数:
262
简介:
Python机器学习算法技术博客，有原创干货！有code实践！ 【更多内容敬请关注公众号 "算法进阶"】
rasbt/MachineLearning-QandAI-book12,300
语言:
Jupyter Notebook
星标数:
973
分叉数:
190
简介:
Machine Learning Q and AI book
pixegami/rag-tutorial-v212,325
语言:
Python
星标数:
969
分叉数:
605
简介:
An Improved Langchain RAG Tutorial (v2) with local LLMs, database updates, and testing.
Denis2054/Transformers-for-NLP-2nd-Edition12,326
语言:
Jupyter Notebook
星标数:
968
分叉数:
361
简介:
Transformer models from BERT to GPT-4, environments from Hugging Face to OpenAI. Fine-tuning, training, and prompt engineering examples. A bonus section with ChatGPT, GPT-3.5-turbo, GPT-4, and DALL-E including jump starting GPT-4, speech-to-text, text-to-speech, text to image generation with DALL-E, Google Cloud AI,HuggingGPT, and more
TingNie/Coursera-ML-using-matlab-python12,331
语言:
Jupyter Notebook
星标数:
967
分叉数:
364
简介:
coursera吴恩达机器学习课程作业自写Python版本+Matlab原版
llSourcell/ChatGPT_Trading_Bot12,347
语言:
Jupyter Notebook
星标数:
965
分叉数:
490
简介:
This is the code for the "ChatGPT Trading Bot" Video by Siraj Raval on Youtube
关键参数与规格
关键参数与规格
参数值
语言Python
类别Tutorials
子类别AIE roadmap
关键词LocalAI, RAG
星标数971
分叉数205
贡献者4
创建日期2023-07-06
更新日期2026-10-14
1 天星标数0
7 天星标数-1
排名12,311
上次排名18,835
排名变化6,524
7 天星标排名17,408
上次 7 天星标排名17,385
7 天星标排名变化-23
总星标排名11,404
上次总星标排名11,391
总星标排名变化-13
国家Singapore
README 摘录

Running Llama 2 and other Open-Source LLMs on CPU Inference Locally for Document Q&A Clearly explained guide for running quantized open-source LLM applications on CPUs using LLama 2, C Transformers, GGML, and LangChain **Step-by-step guide on TowardsDataScience**: ___ Context Third-party commercial large language model (LLM) providers like OpenAI's GPT4 have democratized LLM use via simple API calls. However, there are instances where teams would require self-managed or private model deployment for reasons like data

privacy and residency rules. The proliferation of open-source LLMs has opened up a vast range of options for us, thus reducing our reliance on these third-party providers. When we host open-source LLMs locally on-premise or in the cloud, the dedicated compute capacity becomes a key issue. While GPU instances may seem the obvious choice, the costs can easily skyrocket beyond budget. In this project, we will discover how to run quantized versions of open-source LLMs

on local CPU inference for document question-and-answer (Q&A). ___ Quickstart Ensure you have downloaded the GGML binary file from and placed it into the folder To start parsing user queries into the application, launch the terminal from the project directory and run the following command: For example, Note: Omit the prepended if you are NOT using Poetry ___ Tools **LangChain**: Framework for developing applications powered by language models **C Transformers**: Python bindings for the Transformer models implemented in C

/C++ using GGML library **FAISS**: Open-source library for efficient similarity search and clustering of dense vectors. **Sentence-Transformers (all-MiniLM-L6-v2)**: Open-source pre-trained transformer model for embedding text to a 384-dimensional dense vector space for tasks like clustering or semantic search. **Llama-2-7B-Chat**: Open-source fine-tuned Llama 2 model designed for chat dialogue. Leverages publicly available instruction datasets and over 1 million human annotations. **Poetry**: Tool for dependency management and Python packaging

___ Files and Content : Images relevant to the project : Configuration files for LLM application : Dataset used for this project (i. , Manchester United FC 2022 Annual Report - 177-page PDF document) : Binary file of GGML quantized LLM model (i. , Llama-2-7B-Chat) : Python codes of key components of LLM application, namely , , and : FAISS vector store for documents : Python script to ingest dataset and generate FAISS vector store : Main Python script to launch the application and to pass user query via command

line : TOML file to specify which versions of the dependencies used (Poetry) : List of Python dependencies (and version) ___ References

常见问题

记录了多少星标和分叉？

kennethleungty/Llama-2-Open-Source-LLM-CPU-Inference 有 971 个星标和 205 个分叉。

仓库排名是多少？

kennethleungty/Llama-2-Open-Source-LLM-CPU-Inference 排名第 12,311。

记录的语言和类别是什么？

kennethleungty/Llama-2-Open-Source-LLM-CPU-Inference 有 971 个星标，语言为 Python，类别为 Tutorials。

哪个仓库排名最接近？

kennethleungty/Llama-2-Open-Source-LLM-CPU-Inference 与 rasbt/MachineLearning-QandAI-book 比较，后者排名第 12,300，有 973 个星标。

README

Running Llama 2 and other Open-Source LLMs on CPU Inference Locally for Document Q&A

Clearly explained guide for running quantized open-source LLM applications on CPUs using LLama 2, C Transformers, GGML, and LangChain

Step-by-step guide on TowardsDataScience: https://towardsdatascience.com/running-llama-2-on-cpu-inference-for-document-q-a-3d636037a3d8

Context

  • Third-party commercial large language model (LLM) providers like OpenAI's GPT4 have democratized LLM use via simple API calls.
  • However, there are instances where teams would require self-managed or private model deployment for reasons like data privacy and residency rules.
  • The proliferation of open-source LLMs has opened up a vast range of options for us, thus reducing our reliance on these third-party providers. 
  • When we host open-source LLMs locally on-premise or in the cloud, the dedicated compute capacity becomes a key issue. While GPU instances may seem the obvious choice, the costs can easily skyrocket beyond budget.
  • In this project, we will discover how to run quantized versions of open-source LLMs on local CPU inference for document question-and-answer (Q&A).

Quickstart

  • Ensure you have downloaded the GGML binary file from https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML and placed it into the models/ folder
  • To start parsing user queries into the application, launch the terminal from the project directory and run the following command: poetry run python main.py ""
  • For example, poetry run python main.py "What is the minimum guarantee payable by Adidas?"
  • Note: Omit the prepended poetry run if you are NOT using Poetry

Tools

  • LangChain: Framework for developing applications powered by language models
  • C Transformers: Python bindings for the Transformer models implemented in C/C++ using GGML library
  • FAISS: Open-source library for efficient similarity search and clustering of dense vectors.
  • Sentence-Transformers (all-MiniLM-L6-v2): Open-source pre-trained transformer model for embedding text to a 384-dimensional dense vector space for tasks like clustering or semantic search.
  • Llama-2-7B-Chat: Open-source fine-tuned Llama 2 model designed for chat dialogue. Leverages publicly available instruction datasets and over 1 million human annotations.
  • Poetry: Tool for dependency management and Python packaging

Files and Content

  • /assets: Images relevant to the project
  • /config: Configuration files for LLM application
  • /data: Dataset used for this project (i.e., Manchester United FC 2022 Annual Report - 177-page PDF document)
  • /models: Binary file of GGML quantized LLM model (i.e., Llama-2-7B-Chat)
  • /src: Python codes of key components of LLM application, namely llm.py, utils.py, and prompts.py
  • /vectorstore: FAISS vector store for documents
  • db_build.py: Python script to ingest dataset and generate FAISS vector store
  • main.py: Main Python script to launch the application and to pass user query via command line
  • pyproject.toml: TOML file to specify which versions of the dependencies used (Poetry)
  • requirements.txt: List of Python dependencies (and version)

References

排名相邻
同类项目