返回资讯中心
外部精选
软件工程
#开发者实时流

AI Is Making Verification the Bottleneck for Companies

Somewhere in your company today, an expert rejected a perfectly plausible answer. A controller caught a revenue figure that counted deferred bookings. A security engineer blocked a release that passed every test but wou…

Hacker News 最新porridgeraisin2026/10/26 · 20 天前30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Somewhere in your company today, an expert rejected a perfectly plausible answer. A controller caught a revenue figure that counted deferred bookings. A security engineer blocked a release that passed every test but would have handed an AI agent write access to the production database. That override, escalation, or reversal is the most valuable data your firm produces. If you record enough of those moments and use them, alongside everything else your firm knows and measures, to train an AI model that learns from each correction, you could, in theory, eliminate the need for managers to route information and coordinate action. In doing so, execution gets turbocharged by automation. You might even be able to eliminate hierarchy altogether. That, at least, is the pitch. Microsoft CEO Satya Nadella calls that feedback system the critical “learning loop” of the AI era. He also insists that enterprises need to own it rather than cede it to the frontier AI labs. Palantir’s CEO Alex Karp has likewise argued that companies and governments will need sovereign AI to preserve their independence. Jack Dorsey and Roelof Botha went further, offering a blueprint for an ambitious “company world model”: an omnipresent and continuously updated model of the entire business that could coordinate work autonomously. The vision is admittedly seductive. AI makes context extremely easy to retrieve, digest, and continuously process the moment new information arises. What once took an army of managers and multiple systems of record can theoretically be handed over to a single well-tuned AI model. The solution also draws on decades of economic thinking about why firms exist and how they organize work. In the 1930s, economist Ronald Coase argued that firms arise to reduce the costs of relying on markets: discovering prices, negotiating separate contracts, and adapting agreements as circumstances change. Firms bring activities inside their boundaries when coordinating them internally costs less than transacting through the market. In the 1970s, business historian Alfred Chandler documented how the “visible hand” of professional management outperformed market coordination in large-scale production and distribution. Their shared insight is that internal coordination can confer an economic advantage. If AI can perform that coordinating work more cheaply and reliably than layers of human managers, the economic case for those layers weakens. Dissolving the hierarchy becomes the next logical step. With AI, leaders see a chance to preserve the benefits of internal coordination while stripping out much of the friction of human-to-human management. The problem is that execution is only half of the story. The other half—and the source of some of that apparent friction—is the work of verification. Hierarchy does more than route information; it also verifies: deciding what information means, what deserves the firm’s attention, which outputs can be trusted, and which are worth generating at all. As the ability to generate text, media, and code becomes increasingly commoditized and abundant, the calculation as to which half creates the most value for companies evolves too. When execution is cheap, verification becomes more valuable, and firms turn into verification factories: institutions capable of properly steering AI-generated output and standing behind the results. This explains why AI has progressed unevenly across domains. Enterprise AI has advanced fastest where verification is relatively easy. In coding, outputs can often be tested rapidly and cheaply, helping explain why it has so quickly become a success story of AI deployment. Leaders under pressure to define an AI strategy need to recognize what its success hinges on: a coherent plan to solve the verification problem. They also need to recognize how they may be giving away their advantage in this emerging landscape by letting expertise erode and building on technology stacks they don’t own. Using the economics of verification as a framework, this piece explains what belongs in a firm’s verification loop, why leaders must retain control of it, and how doing so enables reliable autonomy without giving away the tacit knowledge and expertise that are key to long-term competitive advantage. The Verification Bottleneck Increasingly, what can be measured can be automated. Once the relevant information is codified and success can be measured, an AI model can use that feedback to hill-climb toward the desired outcome. But there is a catch: Any divergence between the context a human brings to a problem and the context available to the machine is a potential source of error. Think about the last time you wrote a prompt, only to realize midway through that you had omitted a critical dimension, constraint, or desired outcome. More often than not, the missing detail was something you understood tacitly and took for granted. This is a key reason machine intelligence can feel so “jagged:” astonishingly capable along measurable dimensions, but surprisingly clumsy everywhere else. Verification is the act of closing the gap between the weights that power an AI model and the neural paths and “weights” that power your own decision-making on the same problem. As AI makes execution increasingly cheap, the ability to properly steer and verify AI outputs better than your competitors will be the differentiator. World-class verification factories rely on two assets: unique ground truth and talent. For some firms, unique ground truth is simply a byproduct of scale and incumbency. If you process most of the world’s transactions in a domain, you observe that slice of the economy with a precision nobody can replicate. For others, it comes from extreme focus. If you have underwritten the same narrow risk for decades, you hold ground truth nobody else has seen. Neither is the prerequisite; unique measurement is. The best firms use measurement to turn market uncertainty into manageable risks, repeatable processes, and consistently better products. But ground truth alone is inert. Data only records what happened; when something falls outside the norm, it takes an expert to decide what it means and how to act on it. When domain experts set objectives and verify AI output, they draw on accumulated experience: past errors, corrections, and lessons learned. That experience is itself a form of measurement, one the AI does not have access to. Some call what remains unmeasured taste, others judgment. But these terms are vague, and their boundaries keep shifting: as model outputs improve, more of what once required human judgment is automated. A more concrete, practical distinction starts with a question: What context lives only inside a human, and what is available to the machine? Everything else follows from this distinction. Whatever isn’t yet, or cannot be, measured becomes a bottleneck to safe deployment. Unfortunately, many firms are adopting AI in ways that erode both of these critical assets, data and talent, at once. Tools that summarize documents, interactions, and context risk flattening divergent views into premature consensus, weakening the independence that makes expert talent valuable. Building on technology stacks the firm doesn’t own and letting its experts’ traces accumulate elsewhere erodes the exclusivity of its proprietary data. Together, these choices weaken the very capability that becomes more valuable as AI improves: the ability to tell whether its output is right. In a recent paper on the economics of verification, my coauthors and I mapped tasks along two dimensions: how cheap they are to automate, and how cheap they are to verify. See more HBR charts in Data & Visuals When a task is cheap to automate and cheap to verify, AI can be trusted to execute on its own without a human in the loop. We called this the “safe industrial” zone. But when automation is cheap while verification depends on human expertise, things get dangerous rather quickly. Here, we expect firms to deploy AI even when its outputs are likely to fall short. This is where a firm either defends its role as a verification factory or quietly gives it up. Catching the flaw in an output that is almost right takes experts with the independence and motivation to challenge the model, and ground truth that exists nowhere else. Right now, AI labs are doing everything they can to bring down the cost of verification across domains by aggressively purchasing specialized data and expert evaluations—even going so far as to buy the internal records of bankrupt companies. Each time they acquire richer digital traces of an economically valuable task, they turn something their models once missed into a capability they can sell. This expansion would be less threatening if firms were actively defending their position as verification factories. Most are doing the opposite: handing the labs the traces that make their tasks verifiable while letting the experts who could still catch a bad answer drift out of the loop. The sequence from there is predictable. One task at a time, the labs learn to verify what only in-house experts could verify before. The firm’s edge shrinks to whatever the labs still cannot measure. It becomes a thin wrapper around their intelligence, paying per token for capabilities it once owned while the labs capture a growing share of the actual value. Don’t Automate Away Your Expertise As firms increasingly buy inference from the same AI labs, their advantage will lie in the difference between the answers those labs provide to everyone and the conclusions they reach using proprietary data and expertise. Without that independent contribution, research suggests firms could rapidly converge toward a monoculture. Think about what managers really do. Good managers are verifiers. Yes, they route information, but in doing so they also determine what it means, what deserves others’ attention, and what meets their quality bar. In making these calls, they often retain a significant advantage even over frontier AI. Anyone with deep experience is a fast simulator of a particular slice of reality: a “world model” made of a person, fine-tuned through years of friction with a specific technology, market, customer base, or regulator. Much of the information behind that model is tacit and unrecorded, learned as much from what failed as from what worked. And in deciding what deserves attention, managers also wield real power within the organization. Now consider ceding many of those responsibilities to AI. An agent could turn sales meetings, support tickets, and internal discussions into a ranked list of product priorities, then use that list to assign work. An experienced engineer hesitates over one feature because it reminds them of a past failure, but they have not yet pinned down the connection. All they write is, “Not sure I’m fully comfortable with this.” The agent records strong customer demand and no concrete technical objection, leaving the hesitation out of its conclusions. A manager may still approve the budget, but the model has already shaped what they see and what they know to question. Handing these decisions to AI is a transfer of power, not simple automation, and it can destroy what makes a firm unique. Lose your company’s edge in what it senses and prioritizes, and everything else falls apart rather quickly. Even an AI system trained on company data is constrained by what has been measured and made available to it. It therefore risks prioritizing what the firm has already seen and codified. This can deliver measurable short-run gains by quickly routing decision-makers to existing solutions, while weakening the firm’s ability to identify and adapt to new, unstructured problems. When AI tools struggle to distinguish an emerging signal from a distraction, they can amplify what is already known at the expense of what is still taking shape, pushing the firm toward premature consensus around the wrong answer. Dep

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →