Apache Spark: stats, trends and alternatives

ray-project/ray

Distributed compute framework for scaling Python AI workloads: data processing, training, tuning and serving.

dask/dask

Scales pandas, NumPy and custom Python code to multi-core machines and clusters.

apache/beam

Unified batch and streaming programming model that runs on Dataflow, Flink or Spark.

apache/flink

Stateful stream processing engine with exactly-once semantics, also capable of batch workloads.

Eventual-Inc/Daft

Distributed Rust-powered DataFrame engine for multimodal data such as images, documents and embeddings.

apache/hadoop

The original big-data stack (HDFS, YARN, MapReduce); still in legacy clusters, rarely chosen for new builds.

Apache Spark