How to Build a Data Platform From Scratch

We wrote an eBook on Data Platform Fundamentals to help you build a data platform like the ones happy teams run on. In this book, you’ll learn:

• How composable architectures allow teams to ship faster
• Why data quality matters and how you can catch issues before they reach users
• What observability means and how it will help you solve problems more quickly Download your copy now.

Datalogy: Zephon: Fast, Flexible, Elastically Deterministic Data Loading

Datalogy open-sources Zephon. Zephon applies the classic database virtual-versus-materialized view trade-off to AI training, replacing single-use offline ETL materialization with deterministic, late-binding online execution for stateful $n$-to-$m$ operations like tokenization, bin-packing, and token-level mixing. By decoupling pointer-based curriculum scheduling (WorkSource) from columnar payload fetching (Parquet, Vortex), data teams can stream training batches directly from standard lakehouse formats without paying the storage and compute tax of re-materializing corpus copies for every curation ablation. https://www.datologyai.com/blog/zephon

Claude: Build live dashboards and animate explainers with Claude

The Claude announcement supporting live dashboards reiterates that the dashboards are here to stay, but they may not sit well with BI tools. Dashboards are structured signals for people to communicate and collaborate, so when a dashboard breaks, you break the organization. https://claude.com/resources/articles/dashboards-and-motion

Hoyt Emerson: The Apache Arrow Ecosystem

Apache Arrow standardized in-memory columnar computing, eliminating the serialization tax between systems and turning monolithic databases into a modular, zero-copy stack of interchangeable storage, execution, and transport layers. The author writes an excellent overview of the Apache Arrow ecosystem. https://thefulldatastack.substack.com/p/the-apache-arrow-ecosystem

Sponsored: AI Modernization Guide

AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift. Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it. Download the free guide

Netflix: Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale

The hardest part of end-to-end observability isn't collecting more telemetry or building better dashboards; it's solving the underlying data-modeling disconnect across isolated operational tools. Netflix addresses this by building an Observability Ontology persisted in QuipuDB—a homegrown distributed RDF quad store built on Rust redb—to unify metrics, deployments, alerts, and incident Slack threads into a single semantic contract. https://netflixtechblog.com/ontology-driven-observability-building-an-end-to-end-knowledge-graph-at-netflix-scale-421392cf5adf

LinkedIn: Scaling Spark and Flink Shuffle at LinkedIn with Apache Celeborn

LinkedIn’s evolution from Magnet to Apache Celeborn highlights a fundamental truth of modern data infrastructure: tying shuffle state to ephemeral compute nodes breaks down at petabyte scale. Moving to a dedicated remote shuffle service backed by NVMe and ZSTD compression cut overall Spark runtimes by 14%, slashed shuffle costs by 90%, and made rolling maintenance practically invisible to running jobs. https://www.linkedin.com/blog/engineering/infrastructure/scaling-spark-and-flink-shuffle-at-linkedin-with-apache-celeborn/ eBay: Inside Minerva: building scalable observability for eBay’s monetization platform eBay details Minerva, an observability platform engineered to catch localized pricing defects across 2.6 billion listings without putting the real-time fee path at risk. Instead of blocking transactions on remote brokers, the system drops lightweight event IDs into a bounded local ring buffer and defers 250-column context enrichment to an asynchronous ClickHouse pipeline.

Airbnb: CASTLE: training natural language search before the first real query

Airbnb writes about solving the cold-start problem for natural language search before seeing a single user query with CASTLE. This framework synthesizes training queries and relevance labels directly from historical booking sessions. By pairing booked listings with viewed-but-skipped alternatives, the system prompts an LLM to generate seed-guided queries that match the winner, making relevance labels accurate by construction without circular LLM-as-a-judge bottlenecks. https://medium.com/airbnb-engineering/castle-training-natural-language-search-before-the-first-real-query-3818ef8b4828

Grab: GrabMart data for faster disease outbreak detection

Grab outlines an early-stage conceptual pipeline for retail syndromic surveillance, mapping over-the-counter GrabMart purchases to symptom classes to spot anomalous demand spikes before clinical case reports arrive. To separate genuine outbreak signals from retail noise like promotions or panic buying, the architecture aggregates counts across coarse geographic and time windows against seasonal baselines. https://engineering.grab.com/grabmart-data-for-faster-disease-outbreak-detection

Zalando: Automating Data Ingestion with a RAG Flywheel

Zalando writes about how they eliminated the operational bottleneck of manual supplier schema mapping by building a human-in-the-loop RAG flywheel on AWS Bedrock and PostgreSQL (pgvector). The system combines zero-shot LLM deduction for cold starts with HNSW vector similarity search over historically approved mappings, ranking matches by cosine similarity and human approval frequency to prevent drift. https://engineering.zalando.com/posts/2026/10/structure-automating-data-ingestion-with-a-rag-flywheel.html All rights reserved, Dewpeche Private Limited. I provide links for informational purposes and do not endorse them. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.