Data Engineering Weekly #289

How to Build a Data Platform From Scratch We wrote an eBook on Data Platform Fundamentals to help you build a data platform like the happy teams running on a single platform. In this book, you’ll learn:

• How composable architectures allow teams to ship faster
• Why data quality matters and how you can catch issues before they reach users
• What observability means and how it will help you solve problems more quickly Download your copy now. Editor’s note: Talk on Knowledge Spine My talk on implementing a knowledge spine, which details building the ontology, knowledge graph, and semantic layer, how everything is interconnected, and how to get started, is now live. If you’re implementing an ontology, I would love to hear what strategy you’re using to build order in chaos. Archer: Jev’s Architecture Unmasked Jev is the talk of the town for the last couple of weeks. The author probes TypeSafe’s Jev API to distinguish observed behavior from speculation about its design. The tests suggest Jev processes shared context once, handles decision questions in parallel, and returns probabilities directly instead of generating explanatory text. This could make structured AI decisions faster and cheaper, though the proposed architecture remains an independent inference rather than a confirmed disclosure. https://archerhume.com/posts/jevs-architecture-unmasked/ Astronomer: What Jev will do to data engineering We'll discuss what Jev means for data engineering in the coming weeks. Astronomer writes about using Jev to add reliable judgment columns—such as customer persona, intent, or spam status—to data pipelines without building a custom ML model. Jev returns a typed label and confidence score, letting an Airflow workflow accept high-confidence answers while routing uncertain ones to a stronger model or a person. https://www.astronomer.io/blog/what-jev-will-do-to-data-engineering/ Sponsored: Drive Fund Admin Data Quality In this guide, discover how lean private market firms can:
• Drive high-quality fund admin data without scrambling every quarter
• Get to clean data faster
• Use that data as a vital strategic differentiator Get the Framework DoorDash: Inside Vera, DoorDash’s Data Agent DoorDash writes about building Vera, an internal agent that answers business questions across fragmented data and knowledge sources. Vera retrieves only vetted tables and documents, uses a model of how those sources connect, and improves through evaluations reviewed by the people who own each business domain. The design prioritizes trustworthy answers over unrestricted access, showing that data context and feedback loops matter as much as the model. https://careersatdoordash.com/blog/inside-vera-doordashs-data-agent/ Pinterest: Partition Finalization in Pinterest’s Next-Generation DB Ingestion Framework Pinterest writes about telling downstream jobs when a time-based slice of streaming data is complete enough to use. Its ingestion system records event-time statistics in each Iceberg commit and advances a one-way “safe to read” marker only when it is confident earlier data is unlikely to change. The design lets consumers balance freshness against completeness without relying on a separate coordination service. https://medium.com/pinterest-engineering/partition-finalization-in-pinterests-next-generation-db-ingestion-framework-4c7da6e4cc8f Sponsored: AI Modernization Guide AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift. Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it. Download the free guide Uber: Taming the ML Firehose: Scaling Feature Consistency Uber writes about preventing models from learning from a different version of a feature than the one used during live predictions. Its feature-logging system records the actual inputs used to score selected impressions and pairs them with later outcomes, so the next training set starts from the live record. This removes a common source of model drift while selective logging keeps the data volume manageable. https://www.uber.com/us/en/blog/taming-ml-firehose/ Fresha: Fresh Data Isn’t Fast Data: How to make CDC-backed ML inference fast in Snowflake Fresha writes about getting predictions for newly signed-up businesses to ad platforms sooner, without running its shared analytics models more often. It computes first-day inputs when the hourly scoring job runs, using continuously updated database changes in Snowflake instead of daily prepared tables. This makes timely predictions practical without adding a separate serving system, as long as it can efficiently skip old data. https://medium.com/fresha-data-engineering/fresh-data-isnt-fast-data-how-to-make-cdc-backed-ml-inference-fast-in-snowflake-9a301004270b Red Hat: Data liberation - Apache Kafka’s native cluster mirroring Red Hat writes about making Kafka data easier to copy between separate clusters for recovery and upgrades. The new built-in mirroring lets destination servers fetch and store the source’s records unchanged, keeping message positions and consumer progress aligned. This removes much of the coordination normally needed to switch clusters, helping applications resume with less disruption. https://developers.redhat.com/articles/2026/09/22/data-liberation-apache-kafka-native-cluster-mirroring Deliveroo: Roonomics - How Economics Helped Us to Build a Churn Prediction Model Deliveroo writes about replacing a broad restaurant-churn alert system that overwhelmed retention teams with partners unlikely to leave. Its new model ranks restaurants by their likelihood of churning and focuses outreach on the highest-risk, most important partners. This helps teams spend limited time where intervention is most likely to matter. https://deliveroo.engineering/2026/09/22/roonomics-how-economics-helped-us-to-build-a-churn-prediction-model.html Wayfair: When Assigned Isn’t Exposed: How Wayfair Uses Trigger Analysis to Recover Hidden Test Signals Wayfair writes about experiments that can appear ineffective because many assigned customers never see the change being tested. Its trigger analysis compares only customers who had a real opportunity to encounter the feature, applies the same condition to both test groups, and checks for bias. This helps teams distinguish a genuinely neutral result from a useful change whose signal was hidden by broad measurement. https://www.aboutwayfair.com/careers/tech-blog/when-assigned-isnt-exposed-how-wayfair-uses-trigger-analysis-to-recover-hidden-test-signals All rights reserved, Dewpeche Private Limited. I provide links for informational purposes and do not endorse them. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.