返回资讯中心
外部精选
大数据
#数据平台#湖仓

Do We Still Need Data Models?

As agents build data systems, models must help us understand the decisions behind them.

Data Engineering WeeklyAnanth Packkildurai2026/10/28 · 今天25 分钟阅读

以下正文同步自 Data Engineering Weekly,版权归原站所有,已转换为易读排版。

Agents are changing how we build software. A developer can describe a problem, give an agent access to a codebase, and ask it to investigate, write code, run tests, and propose a change. The developer still needs to review the work, but the path from an idea to a working implementation is getting shorter. I have been thinking about what this means for data engineering. In my earlier piece on the self-organizing data stack, I sketched out what an Agentic Data Development Lifecycle, or ADDLC, could look like. Since then, one question has come up repeatedly: if agents can understand data and build pipelines, do we still need data models? I believe we do. But I also think we need to reconsider what we expect a data model to do. Today, a model helps people translate business requirements into a data system design. In an ADDLC, an agent may take on more of that design and implementation work. People will still need a way to understand what the agent is building, how it interpreted the request, and which decisions it made along the way. That brings me back to a subject that has always fascinated me: how humans and machines communicate.

Signals and language

A red traffic light tells a driver to stop. A monitoring alert tells an engineer that something needs attention. A purchase event tells a downstream system that a transaction occurred. Each is a signal. Its usefulness depends on how the receiver interprets it. Language lets us express more complex ideas. We use it to describe relationships, explain conditions, question assumptions, and negotiate meaning. Even when two people speak the same language, they can hear the same sentence and understand it differently. Data engineering deals with both signals and language. Pipelines move records. Systems emit events and alerts. Schemas define the shape of information moving between them. People use requirements, conversations, diagrams, and code to explain what that information means. They discuss what makes someone a customer, which transactions count as sales, and how the business changes over time. A purchase event can arrive on time, match its schema, and pass every quality check. It still cannot tell us, by itself, whether that purchase makes someone an active customer. To answer that question, we need a shared understanding of “active,” “customer,” and the period we are talking about. We also need a way to carry that understanding into the system. Data modeling has long helped us do that.

What a data model does today

In a human-led development process, a business expert explains a need. A modeler or engineer asks questions, studies the available data, and works with others to turn that need into definitions, relationships, and rules. The team then implements those decisions in schemas, pipelines, and analytical systems. The model gives the team something to discuss as it designs the system. Consider a sales dataset. What does one row represent: an order, an order line, or a payment? Should a returned item reduce sales on the original purchase date or on the return date? If a customer changes their address, do we need the current address or the address at the time of purchase? The answers affect how we organize the data and what people can do with it. This is why I have never thought of a data model as just a diagram or a database design. It captures decisions about the business. The conversation around it can expose assumptions that might otherwise remain buried in code. Kimball’s dimensional design process offers a good example. It starts with the business process and grain, then moves to dimensions and facts. Those choices connect what the business wants to measure with how the data will be organized for analysis. That approach remains useful when an agent builds the system. Someone—or something—still needs to decide what a row represents.

When an agent builds the system

Imagine a marketing leader asks an agent for a dataset of customers who responded to an email campaign. The agent examines the existing data platform and finds a definition of response: a customer clicked on an email. It uses that definition to design the dataset, generates the pipeline, and runs the tests. The pipeline works. The records arrive. The tests pass. But the marketing team wanted to know how many customers purchased within seven days of receiving the email. The agent applied a valid definition, but it answered the wrong question. Now imagine that before building the pipeline, the agent shows the team how it interpreted the request:

  • A response means a qualifying purchase within seven days of email delivery.
  • The dataset links campaign recipients to purchases using a specified customer identifier.
  • An existing click-based definition measures engagement and applies to a different use case.
  • The marketing team needs to decide how to treat returned purchases. The team can inspect those choices, correct the agent’s interpretation, and settle the open question. The agent can then carry the agreed definition into the design and implementation. This exchange gives the model another role. It helps people understand the system the agent proposes to build, not just the structure in which the data will be stored and analyzed. The agent might still use a dimensional model, with purchase facts and customer, campaign, and date dimensions. But the marketing leader reviewing the work may need a different view: what counts as a response, which customers and purchases are included, and which assumptions remain unresolved. Both views matter. The dimensional design helps the team organize the data for analysis. The business-facing view helps the team examine the meaning the agent plans to implement. They need to stay connected.

What humans need to see

If agents do more development work, people will need to see more than finished tables or a success message. They need to understand where an agent found a definition and whether it applies to the current request. They will need to distinguish a relationship the agent inferred from one a domain owner confirmed. They will also need to know when the agent could not find enough information to make a decision. In the campaign example, profiling might suggest that a recipient ID matches a customer ID. That is a useful starting point. It is not the same as a confirmed identity rule. If returned purchases have no clear treatment, the agent should leave that question visible rather than quietly choosing an answer. The explanation also needs to reflect the system that actually gets built. An agent can produce a plausible account of its work and still implement something different. Teams will need ways to check the definitions and decisions against the code, data, and tests—not just read the agent’s summary and assume they match. Context matters beyond a single request. Marketing and customer support may use different definitions of an active customer because they are answering different questions. A definition may change over time or apply only to one channel, product, or region. An agent working across those boundaries needs to know which definition applies and where it came from. In our campaign example, another distinction matters: a purchase within seven days tells us the purchase followed the email. It does not prove that the email caused the purchase. If the team wants to measure incremental campaign impact, it needs a method that can answer that different question. These are the kinds of distinctions that people and agents need to communicate as they work together.

Will we need a new modeling technique?

I don't think ADDLC will require everyone to abandon Kimball or adopt a new modeling technique. The choice will depend on the data, the workload, and the question the team needs to answer. Dimensional modeling can continue to support analytical design. Semantic definitions can make business terms and metrics reusable. An ontology can help describe concepts and relationships across a domain. Other context—such as ownership, evidence, and a decision's history—can help people and agents judge how to use those definitions. These capabilities can work together, but they are not all required for every problem. A team may be able to start with an existing dimensional model and a small set of clear business definitions. Another organization may need a broader ontology because its agents must work across many connected domains. Palantir’s AIP architecture is one example of a platform that uses an ontology to connect data, logic, actions, and security for humans and agents. It is one approach to the problem. The broader question is how we give an agent enough business context to work with while giving people a way to understand and challenge its decisions. Ontologies are not new, and they take effort to build and maintain. Agents may give organizations new reasons to invest in them, but the investment still needs a clear purpose. Someone must own the definitions, resolve disagreements, and keep them current as the business changes. The human-facing model I describe may not be completely separate. It could be a view of the same definitions and decisions that guide the agent’s work, presented in a form people can inspect. We are still learning what that view should look like. So, do we still need data models? Yes. An agent still needs to organize data in a way that supports the work it is building. People still need to agree on what that data means. What may change is the conversation around the model. A human-led team uses it to work through decisions before and during implementation. In an ADDLC, the model may also need to show us how an agent understood a request, what it inferred, and where it needs help. The faster we can turn a request into a working pipeline, the more valuable it becomes to understand the decisions made along the way. Data modeling has always helped people have that conversation. Now we need to make room for agents in it.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →