Daft

Apache Spark

apache/spark

Popular

The standard engine for large-scale data processing, SQL, streaming and ML across clusters.

Ray

ray-project/ray

Popular

Distributed compute framework for scaling Python AI workloads: data processing, training, tuning and serving.

Dask

dask/dask

Scales pandas, NumPy and custom Python code to multi-core machines and clusters.

Apache Beam

apache/beam

Unified batch and streaming programming model that runs on Dataflow, Flink or Spark.

Apache Flink

apache/flink

Trending (detected automatically)Popular

Stateful stream processing engine with exactly-once semantics, also capable of batch workloads.

Apache Hadoop

apache/hadoop

Legacy

The original big-data stack (HDFS, YARN, MapReduce); still in legacy clusters, rarely chosen for new builds.