Apache Flink

Apache Spark

apache/spark

Popular

The standard engine for large-scale data processing, SQL, streaming and ML across clusters.

Ray

ray-project/ray

Popular

Distributed compute framework for scaling Python AI workloads: data processing, training, tuning and serving.

Dask

dask/dask

Scales pandas, NumPy and custom Python code to multi-core machines and clusters.

Apache Beam

apache/beam

Unified batch and streaming programming model that runs on Dataflow, Flink or Spark.

Daft

Eventual-Inc/Daft

Rising

Distributed Rust-powered DataFrame engine for multimodal data such as images, documents and embeddings.

Apache Hadoop

apache/hadoop

Legacy

The original big-data stack (HDFS, YARN, MapReduce); still in legacy clusters, rarely chosen for new builds.

Apache Kafka

apache/kafka

Popular

Distributed event-streaming platform; Kafka 4 removed ZooKeeper in favor of KRaft.

Confluent Cloud

confluent.io

Popular

Fully managed Kafka and Flink service from Kafka's creators, with connectors and governance.

Redpanda

redpanda-data/redpanda

Kafka API-compatible streaming engine in C++ with no JVM or ZooKeeper.