Data Engineering Weekly #290

How to Build a Data Platform From Scratch We wrote an eBook on Data Platform Fundamentals to help you build a data platform like the happy teams running on a single platform. In this book, you’ll learn:

• How composable architectures allow teams to ship faster
• Why data quality matters and how you can catch issues before they reach users
• What observability means and how it will help you solve problems more quickly Download your copy now. Sebastian Raschka: Language Models for Text Classification: From Bag-of-Words to Jev The author writes about the trade-off between flexible language models and custom classifiers that require task-specific training. Jev takes text, a question, and defined answer choices, then returns a label and confidence score without needing a new model for each task. It could make routine routing and screening cheaper to deploy, though high-volume or specialized work may still justify custom models. https://magazine.sebastianraschka.com/p/classifier-history-and-jev Sreedeepsinha: Decision Models Meet LLMs: ai_decide vs ai_classify for PII Checks and LLM Routing The author writes about putting a decision step before a customer-support model, so personal data and request difficulty are assessed deliberately rather than by accident. The author tested the ai_decide’s ability to answer multiple fixed questions with probabilities in one call, allowing messages to be masked or sent to a fitting model. Saving those decisions makes routes auditable and lets uncertain or failed checks default to the safer path. https://medium.com/@sreedeepsinha/decision-models-meet-llms-ai-decide-vs-ai-classify-for-pii-checks-and-llm-routing-8b52cf42a8d8 John Bourous: Decisions are a commodity now. Context is not The author explains why cheaper decision models and generative interfaces do not, by themselves, deliver useful personalization: they still need a current picture of each visitor. Its approach collects live behavior, turns it into a compact session summary, and provides it as plain data to whichever model chooses the next page elements. The argument is that fresh context—not the model—is the lasting infrastructure challenge, because outdated visitor profiles lead to irrelevant decisions. https://x.com/jbourous/status/2105688471654338923 Sponsored: AI Modernization Guide AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift. Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it. Download the free guide Uber: Halving the Time: How Uber Eats Rebuilt Its Search Pipeline Uber writes about rebuilding Uber Eats search after years of accumulated inefficiencies made results too slow to appear. The team removed redundant work, ran steps in parallel, and sent a smaller first page while saving the rest for scrolling. These changes halved search latency, helping customers see results sooner while preserving search quality. https://www.uber.com/us/en/blog/uber-eats-search-pipeline/ Gojek: Per-Bank Consumer Lanes: How We Solved Head-of-Line Blocking with the Bulkhead Pattern Gojek writes about preventing a slow or busy bank from delaying transfers to other banks. The team gave each combination of bank, transfer type, and urgency its own queue and dedicated workers. This isolates delays, protects urgent transfers from bulk traffic, and lets the team increase processing capacity only where needed. https://medium.com/gojekengineering/per-bank-consumer-lanes-how-we-solved-head-of-line-blocking-with-the-bulkhead-pattern-ef4220f9ad4f Airbnb: Personalization without user identity Airbnb writes about personalizing recommendations for visitors with little or no search or booking history. Its system uses approximate location from an IP address to draw on combined search and booking patterns from nearby users who consented to data use. This helps replace generic listings and destination suggestions with more relevant options, without requiring a login or individual activity history. https://medium.com/airbnb-engineering/personalization-without-user-identity-8e7891a1d486 WHOOP: CDC at WHOOP: Self-Service Replication for Hundreds of Postgres Tables WHOOP writes about replacing costly database replication that required data engineers to handle table and column changes. Its system records changes continuously, updates analytical copies every four hours, and automatically carries new tables and columns downstream. This reduces computing costs and lets service teams manage replication themselves, while teams needing fresher data can read the change records directly. https://engineering.prod.whoop.com/cdc-at-whoop/ Astronomer: Durable execution with Apache Airflow® Astronomer writes about preventing crashes from forcing long-running Airflow tasks to repeat costly work. Airflow 3.3 can save progress and job details so retries reconnect to running jobs or reuse completed steps, including AI calls. This reduces wasted computing time and recovery costs, though teams still need to ensure that actions are safe to repeat. https://www.astronomer.io/blog/durable-execution-with-apache-airflow/ Josh Rosen: The Agentic Security Stack: Emerging Architectural Patterns The author writes about securing AI coding agents that produce software faster than people can review it. Emerging approaches limit agents’ access, check their tools and instructions, and use independent reviewers and automated testing to find and validate security flaws. These controls aim to protect both the software and the process used to build it as development becomes more automated. https://x.com/JoshARosen/status/2105694949438075117 Prateek Singh: DataFusion on the Apple Silicon GPU: One Optimizer Rule, the Same SQL, Sorts 6.9x to 28.8x Faster The author writes about speeding up large DataFusion sorts using Apple Silicon’s graphics processor. The ArrowMetal extension moves suitable operations to the GPU and leaves others with DataFusion, without changing SQL queries. Tests on one Apple M4 Max showed faster large sorts and lower CPU use, with matching results, suggesting a way to accelerate selected workloads. The analysis piqued my interest in running analytical workloads on the GPU. https://medium.com/data-arkaive/datafusion-on-the-apple-silicon-gpu-one-optimizer-rule-the-same-sql-sorts-6-9x-to-28-8x-faster-e6777fc5dd9f All rights reserved, Dewpeche Private Limited. I provide links for informational purposes and do not endorse them. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.