← Open Source
NVIDIA

cudf-spark

NVIDIA cuDF for Apache Spark plugin - accelerate Apache Spark with GPUs.

InfrastructureOrchestrationData systemsScala
Open on GitHub
Momentum
+1stars in 24 hours+0.1%
1.01k
Stars
311
Forks
+2
This week
91
Contributors
Created 2020-05-14 · Updated 2026-10-16 · #5933 today
Top developers

NVIDIA/cudf-spark is ranked 5,933 with 1,011 stars.

Same-category comparison

NVIDIA/cudf-spark is ranked 5,933 among these 8 peers; MoonshotAI/checkpoint-engine is ranked 5,934.

Same-category comparison
NameRankMetrics
microsoft/Tutel5,914
Language:
C
Stars:
1,027
Forks:
111
Description:
Tutel MoE: Optimized Mixture-of-Experts Library, Support GLM/Kimi/Gpt/DeepSeek/Qwen using FP8/NVFP4/MXFP4.
slkiser/opencode-quota5,915
Language:
TypeScript
Stars:
1,026
Forks:
112
Description:
OpenCode quota & tokens usage with zero context window pollution. Supports OpenCode Go, Cursor, GitHub Copilot, OpenAl, Kimi Code, Alibaba Coding Plan, Chutes Al, Google Antigravity, Z.ai Coding Plan and more.
kaito-project/kaito5,918
Language:
Go
Stars:
1,025
Forks:
187
Description:
Kubernetes AI Toolchain Operator.
dillionverma/llm.report5,923
Language:
TypeScript
Stars:
1,020
Forks:
86
Description:
📊 llm.report is an open-source logging and analytics platform for OpenAI: Log your ChatGPT API requests, analyze costs, and improve your prompts.
MoonshotAI/checkpoint-engine5,934
Language:
Python
Stars:
1,010
Forks:
113
Description:
Checkpoint-engine is a simple middleware to update model weights in LLM inference engines.
whylabs/langkit5,945
Language:
Jupyter Notebook
Stars:
1,001
Forks:
76
Description:
🔍 LangKit: An open-source toolkit for monitoring Large Language Models (LLMs). 📚 Extracts signals from prompts & responses, ensuring safety & security. 🛡️ Features include text quality, relevance metrics, & sentiment analysis. 📊 A comprehensive tool for LLM observability.
sophgo/tpu-mlir5,948
Language:
C++
Stars:
999
Forks:
229
Description:
Machine learning compiler based on MLIR for Sophgo TPU.
routatic/proxy5,962
Language:
Go
Stars:
986
Forks:
134
Description:
route Claude Code requests through multiple upstream providers (OpenCode Go, OpenCode Zen, and AWS Bedrock) with automatic model selection and format transformation.
Key parameters and specifications
Key parameters and specifications
ParameterValue
LanguageScala
CategoryInfrastructure
SubcategoryOrchestration, Data systems
KeywordsData processing, DevOps, Memory management, Optimization
Stars1,011
Forks311
Contributors91
Created date2020-05-14
Updated date2026-10-16
Stars in 1 day1
Stars in 7 days2
Rank5,933
Previous rank18,817
Rank change12,884
7-day stars rank8,310
Previous 7-day stars rank8,380
7-day stars rank change70
Total stars rank11,064
Previous total stars rank11,051
Total stars rank change-13
CountryUSA
README summary

NVIDIA cuDF plugin for Apache Spark NOTE: For the latest stable README. The NVIDIA cuDF plugin for Apache Spark provides a plugin library that leverages GPUs to accelerate processing via the cuDF (CUDA DataFrame) libraries. Documentation on the current release can be found here. To get started and try the plugin out use the getting started guide. Note: The NVIDIA cuDF plugin for Apache Spark was formerly known as the RAPIDS Accelerator for Apache Spark. The RAPIDS name will be

sunset over time. Artifact names will remain the same for now. Compatibility The SQL plugin tries to produce results that are bit for bit identical with Apache Spark. Operator compatibility is documented here Tuning To get started tuning your job and get the most performance out of it please start with the tuning guide. Configuration The plugin has a set of Spark configs that control its behavior and are documented here. Issues & Questions We use github to track bugs, feature

requests, and answer questions. File an issue for a bug or feature request. Ask or answer a question on the discussion board. Download The jar files for the most recent release can be retrieved from the download page. Building From Source See the build instructions in the contributing guide. Integration The cuDF plugin provides some APIs for doing zero copy data transfer into other GPU enabled applications. Currently, we are working with XGBoost to try to provide this integration out

of the box. You may need to disable RMM caching when exporting data to an ML library as that library will likely want to use all of the GPU's memory and if it is not aware of RMM it will not have access to any of the memory that RMM is holding. Qualification and Profiling tools The Qualification and Profiling tools are available in the NVIDIA/cudf-spark-tools repository. Please refer to Qualification tool documentation and Profiling tool documentation

for more details on how to use the tools. Dependency for External Projects If you develop functionality on top of the cuDF plugin (we currently limit support to GPU-accelerated UDFs) we recommend you declare the distribution artifact as a dependency.

Frequently asked questions

How many stars and forks are recorded?

NVIDIA/cudf-spark has 1,011 stars and 311 forks.

What is the repository rank?

NVIDIA/cudf-spark is ranked 5,933.

What language and category are recorded?

NVIDIA/cudf-spark has 1,011 stars, uses Scala and belongs to Infrastructure.

Which repository is closest in rank?

NVIDIA/cudf-spark is compared with MoonshotAI/checkpoint-engine, ranked 5,934 with 1,010 stars.

README

NVIDIA cuDF plugin for Apache Spark

NOTE: For the latest stable README.md ensure you are on the main branch.

The NVIDIA cuDF plugin for Apache Spark provides a plugin library that leverages GPUs to accelerate processing via the cuDF (CUDA DataFrame) libraries.

Documentation on the current release can be found here.

To get started and try the plugin out use the getting started guide.

Note: The NVIDIA cuDF plugin for Apache Spark was formerly known as the RAPIDS Accelerator for Apache Spark. The RAPIDS name will be sunset over time. GitHub links from spark-rapids will redirect to cudf-spark. Artifact names will remain the same for now.

Compatibility

The SQL plugin tries to produce results that are bit for bit identical with Apache Spark. Operator compatibility is documented here

Tuning

To get started tuning your job and get the most performance out of it please start with the tuning guide.

Configuration

The plugin has a set of Spark configs that control its behavior and are documented here.

Issues & Questions

We use github to track bugs, feature requests, and answer questions. File an issue for a bug or feature request. Ask or answer a question on the discussion board.

Download

The jar files for the most recent release can be retrieved from the download page.

Building From Source

See the build instructions in the contributing guide.

Testing

Tests are described here.

Integration

The cuDF plugin provides some APIs for doing zero copy data transfer into other GPU enabled applications. It is described here.

Currently, we are working with XGBoost to try to provide this integration out of the box.

You may need to disable RMM caching when exporting data to an ML library as that library will likely want to use all of the GPU's memory and if it is not aware of RMM it will not have access to any of the memory that RMM is holding.

Qualification and Profiling tools

The Qualification and Profiling tools are available in the NVIDIA/cudf-spark-tools repository.

Please refer to Qualification tool documentation and Profiling tool documentation for more details on how to use the tools.

Dependency for External Projects

If you develop functionality on top of the cuDF plugin (we currently limit support to GPU-accelerated UDFs) we recommend you declare the distribution artifact as a provided dependency.


    com.nvidia
    rapids-4-spark_2.12
    26.12.0-SNAPSHOT
    provided

Nearby in the ranking
Similar items