Data Engineering Weekly #291

How to Build a Data Platform From Scratch We wrote an eBook on Data Platform Fundamentals to help you build a data platform like the ones happy teams run on. In this book, you’ll learn:

• How composable architectures allow teams to ship faster
• Why data quality matters and how you can catch issues before they reach users
• What observability means and how it will help you solve problems more quickly Download your copy now. Event Alert: NEXT QUERY 2026 | Free Virtual Data + AI Summit | Nov 4 On November 4, NEXT QUERY 2026 brings together engineers from Apple, AWS, Microsoft AI, Intuit, Coinbase, AppLovin, and more to share what worked, what broke, and what they'd do differently. Expect practical lessons from production on: 🤖 AI agents: Business semantics and data relationships for accurate answers. 🔐 Fresh context & governance: Real-time data, access controls, and workload isolation. ⚡ Performance at scale: Fast, reliable, and resilient analytics. 🏗️ Architecture trade-offs: Deployment, migration, performance, and cost. Can’t attend live? Register anyway, and we’ll send you access to the recordings afterward! Explore the agenda and register for free: https://nextquery.phoenixdata.ai/ Booking.com: How We Migrated Europe’s Largest Data Lake: On Time and Without Turning the Lights Off In data engineering, the migration is often the hardest thing. Booking.com writes about moving its data from aging systems to the cloud while keeping daily operations running. The team adopted an incremental approach, moving the workload in small batches and using continuously updated copies of shared data so teams could progress without waiting for one another. The migration kept the business running and gave teams more flexible systems and stronger controls over data access. https://medium.com/booking-com-development/how-we-migrated-europes-largest-data-lake-on-time-and-without-turning-the-lights-off-61ee8a07db76 Affirm: Building Affirm’s Intelligence Layer Affirm writes about creating a shared record of customer activity so each new model does not have to piece together its own data. The system links events such as loan changes and repayments to the right customer and loan, using common definitions and data-quality checks. This gives teams a reliable history to build on, so they can focus on improving decisions instead of repeatedly preparing the same information. https://medium.com/affirmengineering/building-affirms-intelligence-layer-67139cbe44ed Sponsored: Redefining the LP-GP Experience in Private Markets How can a unified data foundation enhance the LP-GP experience? This Ebook explores how private market firms are using unified, AI-enabled data infrastructure to:
• Improve transparency
• Accelerate reporting
• Strengthen investor confidence Download the Ebook Matt Klein: Announcing blob-stream: a Kafka alternative for no fuss, low cost high volume streaming Systems around object storage as a diskless model are increasingly adopted. introduced blob-stream, an open-source, diskless streaming alternative to Kafka that drastically reduces operational overhead and cross-AZ networking costs by utilizing stateless brokers, Amazon S3, and DynamoDB. By intentionally abandoning Kafka API compatibility and strict global ordering, it offers a massively simplified, highly cost-effective architecture tailored for high-throughput cloud workloads. https://blog.bitdrift.io/post/blob-stream-kafka-alternative Victor Coustenoble: After 25 Years in Data, I Only Bet on One Thing: Optionality In data architecture, “it depends” is practically the universal law. The author reflects this in the article: the right approach depends on a company’s mix of legacy systems, cloud maturity, and business goals; locking into a single monolithic path is the biggest risk. https://www.linkedin.com/pulse/after-25-years-data-i-only-bet-one-thing-optionality-coustenoble-uvsqe/ LinkedIn: Inside LinkedIn’s data context layer: curating trustworthy context for humans and agents Context Engineer is the new name for the Analytical Engineer—a role once called Data Engineer. LinkedIn writes about bridging the gap between static catalogs and AI agents by combining automated runtime query mining with Git-managed, human-curated operational knowledge. By organizing this “Context as Code” into a hierarchical index navigable via Model Context Protocol, the system prevents context-window bloat while boosting its AI agents’ analytical accuracy. https://www.linkedin.com/blog/engineering/infrastructure/inside-linkedins-data-context-layer-curating-trustworthy-context-for-humans-and-agents Sponsored: AI Modernization Guide AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift. Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it. Download the free guide Salesforce: Apache Spark Resource Optimization: Lessons From Eight Million Jobs a Day Salesforce writes about finding wasted computing capacity in Spark jobs that finish successfully but use far less than they request. Its platform collects performance data through a shared submission gateway, then compares complete runs to identify jobs that can safely start with fewer resources. Teams can cut costs while checking that run time, failures, and reliability stay acceptable. https://engineering.salesforce.com/apache-spark-resource-optimization-lessons-from-eight-million-jobs-a-day/ Pinterest: From Activity to Intent: Generating User Journeys with LLMs Pinterest writes about moving beyond individual clicks to understand the longer-term projects users are pursuing. A fine-tuned language model reads a person’s recent activity and produces a ranked list of specific journeys, such as planning a kitchen renovation. This helps recommendations and notifications support meaningful goals while filtering out passing interests and unrelated activity. https://medium.com/pinterest-engineering/from-activity-to-intent-generating-user-journeys-with-llms-9652b9417626 All rights reserved, Dewpeche Private Limited. I provide links for informational purposes and do not endorse them. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.