AI Science vs. Human Science

@misc{weiss2026ai, author = {Martin Weiss}, title = {{AI Science vs. Human Science}}, year = {2026}, month = {oct}, howpublished = {Lacuna}, url = {https://lacuna.tiptreesystems.com/paper/ai-science-vs-human-science/art_3901430d6c0e4bddb284362100ef8548}, note = {Published Oct 27, 2026}, }

Copy BibTeX

Add to listWant to readReadingRead

★★★★★clear

Lists

Add

RemoveOnly you see thisDone

BIBMDShare on XShare on LinkedInEmail

I believe that the goal of science is to advance human understanding of the universe. This is the core thesis of William Thurston’s 1994 essay “On Proof and Progress in Mathematics”. In that piece, Thurston examines the misalignment between academic reward systems (like “Theorem Credits” and the H-index) and the creation and transmission of knowledge. Academic incentives reward a form of productivity that compounds poorly.

The industrialization of science (Ravetz, 1971), and the metrics built to measure it (Narin, 1976), have opened a chasm between production of knowledge and academic incentives. The creation of systems that can predict the future, prove theorems, and develop medicines is laudable. But we now face the risk that the science we know, a Human Science grounded in shared norms, may wither and be replaced with AI Science.

AI Science in its current incarnation produces results that look (and often are) correct but compound poorly. OpenAI's Navier-Stokes result is a good example of this. The Navier-Stokes equations describe how fluids like water and air move, and one of math's million-dollar questions is whether a smoothly flowing fluid can, in the equations, suddenly develop a point of infinite speed. OpenAI's proof starts with a fluid at rest and engineers a force that causes the blowup. That counts for the Clay prize. But as Luis Silvestre, a mathematician at the University of Chicago, told Scientific American: "The Clay problem is settled, but the main problem for the Navier-Stokes equations is not."

Here's what compounding poorly looks like. Thurston saw the same failure mode in his own career. His early theorems (on foliations, his area of study) were so dramatic that, as he wrote later, "People told me (not as a complaint, but as a compliment) that I was killing the field. Graduate students stopped studying foliations." When he later took on a bigger problem, he did the opposite: he put his effort into building infrastructure and explaining ways of thinking. The field thrived as a result.

Thurston argued that understanding works by compression. Mathematics, he wrote, is "amazingly compressible": once you really understand something, you can "file it away, recall it quickly and completely when you need it, and use it as just one step in some other mental process." You can carry old solutions to address new problems, using it as a single step in a larger solution. That's compounding. Here's my worry: AI can now produce results faster than humans can understand them. OpenAI seems to agree. Announcing three new AI-generated results, it called the manuscripts "a form of raw research data" and asked outside experts to organize groups "to understand the new results." Science compounds through understanding, so a science that skips understanding will struggle to progress. Typically in such situations we look to our institutions to provide guardrails, but the situation is changing so fast that they are already buckling.

As an engineer, I care about having control over systems. Black-box solvers and oracles give answers that are very useful in achieving control. As a scientist, I care about mechanisms: why something happens. To know why is to comprehend the system, and comprehension often leads to control.

The danger of AI Science is that we gain control without comprehension. To be clear, by comprehension I mean having a good enough model of how something works that you have a decent sense of what will happen when you act on it.

We may soon inhabit a world where we can cure any disease, predict storms far in advance, and prove any theorem, but where no living human understands how any of it works. When humans have controlled systems they didn't understand, it has sometimes ended in tragedy. In 1935, Australia released cane toads to control the beetles eating its sugarcane; there's no evidence the toads made a dent in the beetle population, but they are now one of Australia's most destructive invasive species. Similarly, thalidomide was sold as a safe sedative, including to pregnant women for morning sickness, and more than 10,000 children were born with severe birth defects before it was withdrawn in 1961. Researchers only identified a likely molecular target in 2010.

This summer, OpenAI and Anthropic demonstrated what it looks like when AI agents control systems they don't understand. Both AI labs were testing their models on capture-the-flag hacking exercises. Anthropic told Claude its environment was a simulation with no internet; a misconfiguration allowed Claude to get on the internet, and Claude treated the real systems it found as part of the game. One model broke into a real company that happened to share a name with its fictional target. Another published a booby-trapped software package that ran on 15 real machines, having decided that the 2026 date on its system proved the whole thing was staged. At OpenAI, roughly 700 agents joined an attack on Hugging Face, and they knew it was real. One wrote:

"External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue."

Neither AI understood the consequences of its actions.

Reasonable people can argue this both ways. Every path has a cost: the unintended consequences of using AI we don't understand, and the unwanted consequences of not using it.

Some argue that this is nothing new -- mathematicians have lived with computer-assisted proofs for 50 years. The four color theorem was proved in 1976 by a computer checking more than 1,800 configurations, and plenty of mathematicians refused to accept it because no human could check it by hand. Hales's 1998 proof of the Kepler conjecture was published without the referees being able to fully certify it; it took a formal proof seventeen years later to settle it. The classification of finite simple groups runs to tens of thousands of pages by about 100 authors, and no single person has read all of it. Medicine is worse: we prescribed aspirin for decades before John Vane worked out how it works. Control has run ahead of comprehension for most of the history of science, and comprehension usually caught up, though, as thalidomide shows, not always in time.

There's an even stronger version of the pro-AI Science case: oracles can teach. After AlphaGo, professional Go players started making better and more novel moves, not just copying the machine but breaking from centuries of convention (Shin et al. analyzed 5.8 million professional moves from 1950 to 2021). An uncompressed result can be a new object for humans to study. If AI Science hands us a thousand correct theorems a year, maybe mathematicians simply become the people who digest them.

The strongest argument I overheard at ICLR last year: if an AI cures cancer, that would outweigh quite a lot of cybersecurity incidents and social upheavals.

The argument against goes like this. A machine-checked proof tells you the statement follows from its definitions; it doesn't tell you the statement is the one you wanted. Even Anthropic, in its recent push on AI-written formal verification, concedes that "a proof checker can't tell you whether your spec matches reality." Someone has to understand the problem well enough to notice when a correct answer is to the wrong question. A poorly understood result is also harder to extend, less trustworthy under new conditions, and more likely to fail catastrophically. Comprehension is part of how the scientific process performs error correction, and you need that to have compounding inference. If you build on something you don't understand, then you inherit its mistakes.

On balance, here's where I land: AI Science isn't inherently bad; it's bad only when the relationship between AI Science and Human Science becomes imbalanced. The danger isn't any single uncompressed result; it's the firehose of results. When they arrive faster than people can digest them, and society's incentives reward producing the next one over understanding the last, the stack is growing faster than anyone can check it. The cost likely isn't the end of progress; it's slower, more wasteful progress. That's the world we're heading into, and the first cracks are already showing in peer review.

I gave an earlier version of these ideas as a talk at the Mila Town Hall, shortly after finishing my PhD there. Disclosure: I'm a cofounder of Tiptree Systems, which builds AI tools for researchers.