Data Engineering & Analytics

Schedule, run and monitor data pipelines as dependency graphs of tasks or assets.

Apache Airflow, Dagster, Prefect

Engines for transforming large datasets in parallel across a cluster.

Apache Spark, Ray, Dask

Durable event logs and engines that process data continuously as it arrives.

Apache Kafka, Apache Flink, Confluent Cloud

Model, test and document in-warehouse transformations, and lint or transpile SQL.

dbt, SQLMesh, SQLGlot

Move data from SaaS apps, APIs and databases into warehouses and lakes, in batch or via change data capture.

Fivetran, Airbyte, dlt

Open table formats and catalogs that bring ACID tables and schema evolution to files on object storage.

Apache Iceberg, Delta Lake, Apache Hudi

SQL engines that query data where it lives, and the in-memory and file formats underneath them.

DuckDB, Trino, Apache Arrow

Managed analytical platforms where most companies store and model their business data.

Snowflake, Databricks, Google BigQuery

In-memory data manipulation and numerical computing libraries at the heart of data science.

pandas, NumPy, Polars

Interactive environments for exploring data, prototyping analyses and sharing results.

Jupyter, JupyterLab, Google Colab

Business intelligence platforms for self-serve exploration, dashboards and reporting.

Power BI, Tableau, Looker

Plotting libraries for charts in notebooks, reports and data apps.

Matplotlib, Plotly, seaborn

Test and monitor data, track lineage and help teams discover and govern their data assets.

Great Expectations, Soda Core, Pandera