返回资讯中心
外部精选
软件工程
#开发者实时流

The inextricable link between comprehension and the ability to steer ship

The pace of change and discovery has exceeded even some of our boldest expectations.

Hacker News 最新vinchenkov2026/10/30 · 18 天前30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Share

The pace of change and discovery has exceeded even some of our boldest expectations.

But much of this development appears to be happening without meaningful digestion by people. The increasingly automated AI research feedback loop is only accelerating these rates.

This dynamic might lead some to believe the use of AI will only reduce our understanding of the world around us, and by extension lead to a loss of influence.

What if the end results were instead amplified abilities to discern, shape, and direct the systems that influence us? Yes, sounds fanciful. Neither outcome is inevitable but the second scenario sounds more enticing.

There isn’t a radical departure needed from the process that’s continually improved the human condition. A preservation of discipline to remain thoughtful participants in building, producing, and discovering is perhaps all that is needed. This type of engagement facilitates an understanding that not only informs how we move forward, but where to.

I think a useful premise to frame the conversation going forward is to define and distinguish “evolution” and “progress”.

First let’s define evolution in this context to mean ordered, technological change. AI enables new kinds of systems to be built. Long standing math and physics problems are solved. New drugs are produced.

Nominally these sound great, but they in themselves don’t benefit us without our consideration. I’ll use two examples to elaborate.

A frontier lab challenges an AI to develop a new theory of fundamental physics and run experiments in a sophisticated closed lab. The experiments surface revelatory findings that challenge our models of the universe. What if no one simply bothered to read the findings afterwards? Did physics “progress”?

A less absurd version of this is happening in mathematics right now. OpenAI recently dropped hundreds of results on open research mathematics problems. Some Fields Medalists are struggling with what this means for their careers, their sense of purpose. For unproven undergraduates in the field, especially so.

Many of the OpenAI results include proofs that can run through digital, automated verifiers, so many of them indeed are indeed correct. As to what they correctly claim isn’t exactly known yet. But unlike the physics example, mathematicians will read through these findings.

Still, it might take years for researchers to build useful abstractions from them. Someone might ask, why should humans continue to build abstractions? If AI will continually improve and solve increasingly complex problems for us, then what’s the point?

The FAA is a year into its long-due modernization of the national air traffic control system. Recent staffing issues haven’t helped with the aging system. A contractor, Peraton, has been recruited to facilitate the rebuild with a deadline of 2028 for the first phase. One part of the contract will “penalize unnecessary delays” to ensure efficient tax dollar usage.

I’m not familiar with the contractor’s reputation and a deadline is a reasonable constraint to prevent indefinite upgrade delays to already aging infrastructure. AI will surely be used here to some degree. That in itself isn’t disqualifying.

However, take the hypothetical that the contractor is under increasing external pressures and decides to accelerate their timeline by delegating more development to AI. At first they keep in tandem, they specify outcomes, and procedural checks are passed.

Over time confidence is gained, more powerful AI models are released, and checks continue to pass but complacency settles in.

The changelogs accumulate and human review becomes superficial. The maintainers of the software, whoever they are at that point, become so detached from the machinery of the system that their understanding of it has become sufficiently misrepresentative of what’s happening underneath.

What occurs after that state is reached depends on many things. One thing that we can be sure of is that human judgement of how the machine works is no longer a factor in the system’s reliability. At that point we rely only on rote checks, the mercy of the Old Gods, and the New.

Back to the question of why continuing to build abstractions matters. Not ones that just give us the feeling of being in the loop, but high-fidelity, empowering ones.

The Millennium Prize problems were curated questions whose answers not only move the frontier of mathematics but impact the domains of cryptography, computer science, and physics.

These questions were selected, in part, because each were aspirational in scope and would mobilize researchers in the direction of rich semantic landscapes thought relevant to future human endeavors. Each landscape would act as a jumping off point of what questions to ask next.

This ability to direct discovery is still true today. And for as long as people are in control, it will continue to be. Even if you operate with the premise that AI becomes better at proving theorems and can even ask new bridging questions themselves, where we want to head next is still up to us. To determine where to head next, we still have to know where we currently stand. Even with OpenAI’s latest contributions, the thinking machines can’t know things for you. They can, however, help you think about things.

Progress has always implied an underlying humanity but it making that explicit adds a bit of clarity now.

Progress is human-directed technological and cultural evolution.

Progress implies there’s some destination, some goals. But sometimes those goals aren’t exactly what we wanted.

Let’s go back to the air traffic control system.

Some system thinkers might argue that if we built a highly-varied suite of checks that ensure human prescribed outcomes, then it doesn’t matter if we understand the underlying system as long as it behaves the way we want.

Say we have super smart AI and give it the ambiguous goal of "Make the system ultra-reliable to minimize injuries.”

The system might reply with something like this, at the same level of granularity your request was stated with:

“I’ve identified 116 architectural risks and modified the system to fix them.”

Would this give you assurance? Of course not. Because even if you did think those risks were properly identified and addressed, what implicit tradeoffs were made? All the checks that “ensure” our specified outcomes still passed. But what side-effects were removed, introduced, or modified?

In software engineering, there’s a concept named Hyrum’s Law. However it can be generalized to any system really. A generalized version might state something like this: Given a sufficient amount of users, all observable system behaviors will be depended on by somebody or something, regardless of the system’s intended use.

So, say you want AI to implement a newly declared goal, call it requirement C, to an existing system. Your AI assesses requirement C does not interfere with existing requirements A and B. It accepts the request without fuss and implements C along with various new tests ensuring outcome C. All your requirements are now satisfied and each check is green. Still, there may be people or other applications out there who have arranged their work, their timing, and their expectations around some observable behavior that’s been around for years but was removed implicitly by changeset C.

It’s a bit analogous to “be careful what you wish for”.

From The Role of Software in Spacecraft Accidents, Nancy Leveson suggests:

“Almost all software-related aerospace accidents (and accidents in other industries) have been related to flawed requirements and misunderstanding about what the software should do— the software performed exactly as the designers intended (it did not “fail”), but the designed behavior was not safe from a system viewpoint.”

So how do you account for these unintended consequences? Instead of asking AI to make the system “ultra-reliable”, you think about what you really want. You want planes to have some minimum separation in the skies, you want precise scheduling, automated alerts. You also want to make explicit in the requirements of the system any implicit dependencies, that you know of.

Now, it can be argued that if you keep tacking on and redefining requirements, specifying those outcomes to AI, then eventually your system will behave in the manner you want. I believe this is more or less true.

But what was necessary in order to ideate over those many discrete desired outcomes, especially for the non-functional ones such as reliability?

This forces one to think and experiment meticulously about different scenarios, combinations of inputs and environments, and what effect they should have on the system. It requires a decomposition of high-level goals from stakeholders and dependents to more concrete ones that more closely align with what everybody really wants.

A sort of convergence of comprehension happens. You start to understand the system from a top-down approach. Ideally as you collaborate with AI to implement requirements, its bottom up understanding deepens your mental model of the system. False assumptions are flagged, consequential properties and hidden dependencies are surfaced, all of which may require you to refine your original requirements.

Of course how much you rinse and repeat this before deployment depends on the nature of the system1. An update to a hyperscaler’s cloud infrastructure demands more due diligence than one for a dating app.

So long as we continue as engaged participants in the process of building, as was true in the past because there was literally no alternative, machine intelligence takes the form of an amplifier rather than a force of alienation.

The above makes it seem like we don’t gain much by using AI to push science or to develop our systems. That’s not what I was claiming.

The above serves to demonstrate the perpetual need of our careful thought of what we want from science and the systems that govern our lives, the constant recalibration of our ecosystem’s alignment.

Legibility and comprehension help facilitate this alignment. I think AI’s potential usefulness is obvious here. Many today use it to help model and illustrate sciences, search and synthesize information, and help understand how we should build systems.

I think it’ll be especially useful for our government’s seemingly archaic administrative systems.

We have newfound capability to make them legible. I’ve experienced firsthand how the DMV Investigations office couldn’t run a simple query of what vehicles I’m liable for, lengthening the process of my identity fraud case by months.

Once proper controls are set and authorization is afforded to AI, it can help connect these disparate data sources and remedy the comically difficult to navigate data relationships. We then leverage that legibility to more effectively redesign and overhaul access to services.

Imagine a 78 year old widow talking over the phone to a state administrator, or a 24/7 available bot, that can seamlessly connect her to the social benefits she’s entitled to? Maybe even over the course of a week rather than months.

Our political and legal systems are more abstract than physical ones, but the need for comprehension still applies. Our laws encode certain responsibilities of the state toward the individual and also encode mechanisms by which the individual can advocate against the state and industry. If AI is used uncritically to draft an increasing share of the legislation, the direct interaction and interpretation of law may become increasingly more alien.

Whether these systems become more or less legible depend on if we use AI as a sort of flashlight here or as an autopilot.

They have provided a dramatic new approach of how we can conceptualize things. They can illuminate topics with rich diagrams, visuals, and interactivity. They can also clearly convey many ideas with just text.

At the same time if you’re wo

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →