Affiliations and Partners

Relational and NoSQL staples — PostgreSQL, MongoDB, Redis — remain core. Vector and graph stores such as Neo4j (and emerging vector databases) matter more as LLM workloads need relationship-aware and embedding-native retrieval.

Separating storage from compute — the pattern Amazon S3 and Google Cloud Storage normalised — remains the foundation for scalable, cost-controlled data estates.

Cloud-native estates displaced traditional warehouse-only models. The live contest is lakehouse: Databricks and Snowflake combine lake and warehouse patterns with independently scalable storage and compute. Oracle and peers still appear where estates demand them.

Open formats such as Apache Iceberg and Delta Lake are now table-stakes for portable lakehouse design. Independent benchmarks (including Dremio’s published work) keep performance claims under scrutiny.

Apache Kafka remains the reference for event and stream ingestion. Reverse ETL — syncing processed data back into operational systems — is the complementary motion to watch.

Beyond Airflow and dbt, Prefect and Apache NiFi compete on flexibility and operability for orchestration and flow design.

AWS Lambda and Azure Functions let teams ship data and event logic without owning the runtime — more time on data, less on undifferentiated infrastructure.

As estates grow, quality and observability stop being optional. Great Expectations, Datafold and peers reflect the shift from hopeful pipelines to tested, monitored ones.

Privacy and compliance pressure make catalog and governance platforms essential. Acryl (DataHub), Collibra and Apache Atlas are representative of the layer that makes ownership and lineage operable.

Traditional BI such as Power BI sits alongside specialised log and search analytics — Splunk and Elasticsearch — when operational signal matters as much as the dashboard.

Kubeflow and MLflow represent the move of ML into operated software: experiment tracking, packaging, deployment and lifecycle control inside the wider platform.

Better models start with better data. DVC positions itself as data version control for the GenAI era; Pachyderm focuses on data-driven pipelines for ML — both treat data as a first-class artefact.

Unlike ordinary software, models degrade under data drift and concept drift. Observability — automated monitoring, explainability and proactive maintenance — is how production ML stays honest after launch.