LakeSoul

lakesoul-io/LakeSoul is ranked 8,036 among these 8 peers; av/harbor is ranked 8,038.

Beyond the table format itself, LakeSoul comes with built-in automated disaggregated multi-level compaction, fine-grained RBAC (including S3 proxy-based access control), high-performance OLAP queries, vector retrieval, full-text retrieval, and native multimodal data processing powered by Ray and Daft. Instead of assembling and maintaining separate catalogs, compaction services, and auth layers, you get a production-ready lakehouse out of the box. Whether you're querying via Spark, streaming via Flink, or training models via PyTorch, Ray, or

Daft, every engine and every language shares the same ACID guarantees, the same upsert semantics, and the same read performance. There are no per-language/per-engine re-implementations of the table format, no subtle behavioral divergences between bindings, and no fragmented compatibility matrix to navigate. 5 | ✓ Batch | ✓ Batch | Java / Python / Scala / SQL | | Flink | 1. 20 | ✓ Streaming | ✓ Streaming | Java / SQL | | Presto | 0. 296(velox) | ✓ Batch | - | SQL | | Ray | 2. 55 | ✓ Distributed | ✓ Distributed | Python | | Daft | 0. 0+ | ✓ Standalone | ✓ Standalone | Python | Core Features

LakeSoul is a cloud-native Lakehouse framework that supports scalable metadata management, ACID transactions, efficient and flexible upsert operation, schema evolution, and unified streaming & batch processing. LakeSoul supports multiple computing engines to read and write lake warehouse table data, including Spark, Flink, Presto, PyTorch, Ray and Daft. LakeSoul supports storage systems such as HDFS and S3. LakeSoul supports three physical file format modes: vortex-compact (default), parquet, and vortex. Vortex file format can be used to store multimodal data and

vector embeddings. LakeSoul was originally created by DMetaSoul company and was donated to Linux Foundation AI & Data as a sandbox project since May 2023. LakeSoul implements incremental upserts for both row and column and allows concurrent updates. LakeSoul uses LSM-Tree like structure to support updates on hash partitioning table with primary key, and achieves very high write throughput while providing optimized merge on read performance (refer to Performance Benchmarks). LakeSoul scales metadata management and achieves ACID control by using

PostgreSQL. LakeSoul uses Rust to implement the native metadata layer and IO layer, and provides C/Java/Python interfaces to support the connecting of multiple computing frameworks such as big data and AI. LakeSoul supports concurrent batch or streaming read and write. Both read and write supports CDC semantics, and together with auto schema evolution and exacly-once guarantee, constructing realtime data warehouses is made easy. LakeSoul supports multi-workspace and RBAC. LakeSoul uses Postgres's RBAC and row-level security

policies to implement permission isolation for metadata. Together with the S3 proxy authorization layer, physical data isolation can be achieved. LakeSoul's permission isolation is effective for SQL/Java/Python jobs. LakeSoul supports automatic disaggregated size-tiered multi-level compaction, automatic table life cycle maintenance, automatic data asset statistics, and automatic redundant data cleaning, reducing operation costs and improving usability. More detailed features please refer to our doc page: Documentations Quick Start Follow the Quick Start to quickly set up a

test env. Tutorials Please find tutorials in doc site: Checkout Examples of Python Data Processing and AI Model Training on