AI Rank
中文
← Open Source
kennethleungty

Llama-2-Open-Source-LLM-CPU-Inference

Running Llama 2 and other Open-Source LLMs on CPU Inference Locally for Document Q&A.

TutorialsAIE roadmapPython
Open on GitHub
Momentum
+0stars in 24 hours0.0%
971
Stars
205
Forks
+-1
This week
4
Contributors
Created 2023-07-06 · Updated 2026-10-14 · #12311 today
Top developers
kennethleungtyeltociearmsfasha

kennethleungty/Llama-2-Open-Source-LLM-CPU-Inference is ranked 12,311 with 971 stars.

Same-category comparison

kennethleungty/Llama-2-Open-Source-LLM-CPU-Inference is ranked 12,311 among these 7 peers; rasbt/MachineLearning-QandAI-book is ranked 12,300.

Same-category comparison
NameRankMetrics
Azure-Samples/contoso-real-estate12,289
Language:
JavaScript
Stars:
974
Forks:
529
Description:
Intelligent enterprise-grade reference architecture for JavaScript, featuring OpenAI integration, Azure Developer CLI template and Playwright tests.
aialgorithm/Blog12,292
Language:
Jupyter Notebook
Stars:
974
Forks:
262
rasbt/MachineLearning-QandAI-book12,300
Language:
Jupyter Notebook
Stars:
973
Forks:
190
Description:
Machine Learning Q and AI book.
pixegami/rag-tutorial-v212,325
Language:
Python
Stars:
969
Forks:
605
Description:
An Improved Langchain RAG Tutorial (v2) with local LLMs, database updates, and testing.
Denis2054/Transformers-for-NLP-2nd-Edition12,326
Language:
Jupyter Notebook
Stars:
968
Forks:
361
Description:
Transformer models from BERT to GPT-4, environments from Hugging Face to OpenAI. Fine-tuning, training, and prompt engineering examples. A bonus section with ChatGPT, GPT-3.5-turbo, GPT-4, and DALL-E including jump starting GPT-4, speech-to-text, text-to-speech, text to image generation with DALL-E, Google Cloud AI,HuggingGPT, and more.
TingNie/Coursera-ML-using-matlab-python12,331
Language:
Jupyter Notebook
Stars:
967
Forks:
364
llSourcell/ChatGPT_Trading_Bot12,347
Language:
Jupyter Notebook
Stars:
965
Forks:
490
Description:
This is the code for the "ChatGPT Trading Bot" Video by Siraj Raval on Youtube.
Key parameters and specifications
Key parameters and specifications
ParameterValue
LanguagePython
CategoryTutorials
SubcategoryAIE roadmap
KeywordsLocalAI, RAG
Stars971
Forks205
Contributors4
Created date2023-07-06
Updated date2026-10-14
Stars in 1 day0
Stars in 7 days-1
Rank12,311
Previous rank18,835
Rank change6,524
7-day stars rank17,408
Previous 7-day stars rank17,385
7-day stars rank change-23
Total stars rank11,404
Previous total stars rank11,391
Total stars rank change-13
CountrySingapore
README summary

Running Llama 2 and other Open-Source LLMs on CPU Inference Locally for Document Q&A Clearly explained guide for running quantized open-source LLM applications on CPUs using LLama 2, C Transformers, GGML, and LangChain **Step-by-step guide on TowardsDataScience**: ___ Context Third-party commercial large language model (LLM) providers like OpenAI's GPT4 have democratized LLM use via simple API calls. However, there are instances where teams would require self-managed or private model deployment for reasons like data

privacy and residency rules. The proliferation of open-source LLMs has opened up a vast range of options for us, thus reducing our reliance on these third-party providers. When we host open-source LLMs locally on-premise or in the cloud, the dedicated compute capacity becomes a key issue. While GPU instances may seem the obvious choice, the costs can easily skyrocket beyond budget. In this project, we will discover how to run quantized versions of open-source LLMs

on local CPU inference for document question-and-answer (Q&A). ___ Quickstart Ensure you have downloaded the GGML binary file from and placed it into the folder To start parsing user queries into the application, launch the terminal from the project directory and run the following command: For example, Note: Omit the prepended if you are NOT using Poetry ___ Tools **LangChain**: Framework for developing applications powered by language models **C Transformers**: Python bindings for the Transformer models implemented in C

/C++ using GGML library **FAISS**: Open-source library for efficient similarity search and clustering of dense vectors. **Sentence-Transformers (all-MiniLM-L6-v2)**: Open-source pre-trained transformer model for embedding text to a 384-dimensional dense vector space for tasks like clustering or semantic search. **Llama-2-7B-Chat**: Open-source fine-tuned Llama 2 model designed for chat dialogue. Leverages publicly available instruction datasets and over 1 million human annotations. **Poetry**: Tool for dependency management and Python packaging

___ Files and Content : Images relevant to the project : Configuration files for LLM application : Dataset used for this project (i. , Manchester United FC 2022 Annual Report - 177-page PDF document) : Binary file of GGML quantized LLM model (i. , Llama-2-7B-Chat) : Python codes of key components of LLM application, namely , , and : FAISS vector store for documents : Python script to ingest dataset and generate FAISS vector store : Main Python script to launch the application and to pass user query via command

line : TOML file to specify which versions of the dependencies used (Poetry) : List of Python dependencies (and version) ___ References

Frequently asked questions

How many stars and forks are recorded?

kennethleungty/Llama-2-Open-Source-LLM-CPU-Inference has 971 stars and 205 forks.

What is the repository rank?

kennethleungty/Llama-2-Open-Source-LLM-CPU-Inference is ranked 12,311.

What language and category are recorded?

kennethleungty/Llama-2-Open-Source-LLM-CPU-Inference has 971 stars, uses Python and belongs to Tutorials.

Which repository is closest in rank?

kennethleungty/Llama-2-Open-Source-LLM-CPU-Inference is compared with rasbt/MachineLearning-QandAI-book, ranked 12,300 with 973 stars.

README

Running Llama 2 and other Open-Source LLMs on CPU Inference Locally for Document Q&A

Clearly explained guide for running quantized open-source LLM applications on CPUs using LLama 2, C Transformers, GGML, and LangChain

Step-by-step guide on TowardsDataScience: https://towardsdatascience.com/running-llama-2-on-cpu-inference-for-document-q-a-3d636037a3d8

Context

  • Third-party commercial large language model (LLM) providers like OpenAI's GPT4 have democratized LLM use via simple API calls.
  • However, there are instances where teams would require self-managed or private model deployment for reasons like data privacy and residency rules.
  • The proliferation of open-source LLMs has opened up a vast range of options for us, thus reducing our reliance on these third-party providers. 
  • When we host open-source LLMs locally on-premise or in the cloud, the dedicated compute capacity becomes a key issue. While GPU instances may seem the obvious choice, the costs can easily skyrocket beyond budget.
  • In this project, we will discover how to run quantized versions of open-source LLMs on local CPU inference for document question-and-answer (Q&A).

Quickstart

  • Ensure you have downloaded the GGML binary file from https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML and placed it into the models/ folder
  • To start parsing user queries into the application, launch the terminal from the project directory and run the following command: poetry run python main.py ""
  • For example, poetry run python main.py "What is the minimum guarantee payable by Adidas?"
  • Note: Omit the prepended poetry run if you are NOT using Poetry

Tools

  • LangChain: Framework for developing applications powered by language models
  • C Transformers: Python bindings for the Transformer models implemented in C/C++ using GGML library
  • FAISS: Open-source library for efficient similarity search and clustering of dense vectors.
  • Sentence-Transformers (all-MiniLM-L6-v2): Open-source pre-trained transformer model for embedding text to a 384-dimensional dense vector space for tasks like clustering or semantic search.
  • Llama-2-7B-Chat: Open-source fine-tuned Llama 2 model designed for chat dialogue. Leverages publicly available instruction datasets and over 1 million human annotations.
  • Poetry: Tool for dependency management and Python packaging

Files and Content

  • /assets: Images relevant to the project
  • /config: Configuration files for LLM application
  • /data: Dataset used for this project (i.e., Manchester United FC 2022 Annual Report - 177-page PDF document)
  • /models: Binary file of GGML quantized LLM model (i.e., Llama-2-7B-Chat)
  • /src: Python codes of key components of LLM application, namely llm.py, utils.py, and prompts.py
  • /vectorstore: FAISS vector store for documents
  • db_build.py: Python script to ingest dataset and generate FAISS vector store
  • main.py: Main Python script to launch the application and to pass user query via command line
  • pyproject.toml: TOML file to specify which versions of the dependencies used (Poetry)
  • requirements.txt: List of Python dependencies (and version)

References

Nearby in the ranking
Similar items