Elasticsearch Examples & Apps

This repo contains executable Python notebooks, sample apps, and resources for testing out the Elastic platform across search, security, and observability use cases.

Visit Elastic Labs for the latest articles, tutorials, and more.

• Elasticsearch Labs
• Security Labs
• Observability Labs

What you can build & learn

Elasticsearch:

• Use Elasticsearch as a vector database to store embeddings, run kNN search, and merge vector and keyword results with hybrid search and reciprocal rank fusion (RRF).
• Build retrieval augmented generation (RAG), question answering, and chatbot apps with semantic_text, the inference API, and semantic reranking.
• Get strong semantic search out of the box with ELSER and multilingual E5, without training or fine-tuning a model.
• Tune relevance with query rules, synonyms, and learning to rank.
• Connect Elasticsearch to OpenAI, Hugging Face, Cohere, Amazon Bedrock, Anthropic, and LangChain.

Observability:

• Instrument your services with OpenTelemetry (OTel), including the Elastic Distributions of OpenTelemetry (EDOT), and send traces, metrics, and logs to Elastic.
• Query and analyze logs at scale with ES|QL and other query languages like PromQL.
• Monitor LLM-powered apps, including latency, token usage, and cost.
• Speed up root cause analysis with Elastic Agent Builder and machine learning anomaly detection.

Security:

• Reproduce the detection engineering and threat hunting techniques from Security Labs Threat Command with ES|QL and EQL queries.
• Explore the tooling behind malware analysis and threat intelligence write-ups.
• Test attacks against LLM applications, such as prompt injection, and learn how to detect them.

Watch, fork, or star the Elastic Labs repo on GitHub for the latest updates.

Apps

• Chatbot RAG App
• Internal Knowledge Search
• Relevance Workbench

Python notebooks 📒

The notebooks folder contains a range of executable Python notebooks, so you can test these features out for yourself. Colab provides an easy-to-use Python virtual environment in the browser.

Generative AI

• question-answering.ipynb
• chatbot.ipynb

Playground RAG Notebooks

Try out Playground in Kibana with the following notebooks:

• OpenAI Example
• Anthropic Claude 3 Example

LangChain

• question-answering.ipynb
• langchain-self-query-retriever.ipynb
• Question Answering with Self Query Retriever
• BM25 and Self-querying retriever with elasticsearch and LangChain
• langchain-vector-store.ipynb
• langchain-vector-store-using-elser.ipynb
• langchain-using-own-model.ipynb

Document Chunking

• Document Chunking with Ingest Pipelines
• Document Chunking with LangChain Splitters
• Calculating tokens for Semantic Search (ELSER and E5)
• Fetch surrounding chunks

Search

• 00-quick-start.ipynb
• 01-keyword-querying-filtering.ipynb
• 02-hybrid-search.ipynb
• 03-ELSER.ipynb
• 04-multilingual.ipynb
• 05-query-rules.ipynb
• 06-synonyms-api.ipynb
• 07-inference.ipynb
• 08-learning-to-rank.ipynb
• 09-semantic-text.ipynb

Semantic reranking

• 10-semantic-reranking-retriever-cohere.ipynb
• 11-semantic-reranking-hugging-face.ipynb

Integrations

• loading-model-from-hugging-face.ipynb
• openai-semantic-search-RAG.ipynb
• amazon-bedrock-langchain-qa-example.ipynb
• Semantic Search using the Inference API with the Cohere Service

Model Upgrades

• upgrading-index-to-use-elser.ipynb

Contributing 🎁

See contributing guidelines.

Support 🛟

The Search team at Elastic maintains this repository and is happy to help.

Official Support Services

If you have an Elastic subscription, you are entitled to Support services for your Elasticsearch deployment. See our welcome page for working with our support team. These services do not apply to the sample application code contained in this repository.

Discuss Forum

Try posting your question to the Elastic discuss forums and tag it with #esre-elasticsearch-relevance-engine

Elastic Slack

You can also find us in the #search-esre-relevance-engine channel of the Elastic Community Slack

License ⚖️

This software is licensed under the Apache License, version 2 ("ALv2").