Nature Is the Final Verifier

AI will end up doing most of science, from running labs to pointing telescopes, and a lot of what it finds nobody will understand. In math that's fine, because Lean can check a proof nobody can read. Physics, chemistry and biology don't have a Lean. The only thing that can check them is nature.

You make a prediction, run the experiment, and the measurement agrees or it doesn't. So if AI is going to do science, it has to get into the physical world.

A lot of what AI finds in science, soon nobody will be able to follow, and I'm fine with that.

vixhaℓ said it well:

If we want science to continue advancing, we have to drop the idea that every correct account of nature must stay fully understandable to humans. Reality does not scale itself to the size of our minds. Eventually the accurate description will be too complex, too high-dimensional, or too non-intuitive for any person to properly hold in their mind. That does not make the description invalid. It just means human comprehension can no longer be treated as a requirement. AI is the practical way to work with that level of complexity.

We see it in math already. In September OpenAI said about 10,000 of its agents found a proof that a version of the Navier–Stokes equations can blow up, and verified it in Lean. It isn't peer reviewed yet, and a human still has to confirm that the Lean statement asks the question we meant.

The mathematician Jacob Tsimerman thinks that's where math is going:

I think soon we are going to have proofs no one understands, and they're going to underlie mathematics, and based on that mathematics, other things are going to work. Does a train run even if no one understands the mechanics by which it runs? Yes, it does.

Physics has worked like this for a while. Feynman said nobody understands quantum mechanics, yet every transistor we make is built on it. Brace yourself for the final theory of quantum and relativity to come with a page of constants nobody can explain.

What's left for us is taste. We pick the questions and decide what counts as a discovery, and AI does most of the solving. We'll probably still take the credit :)

Sometimes nature takes a long time to answer. A biology experiment can take a week or two, and a new drug takes years of clinical trials.

I think verification is the last bottleneck. An AI can come up with hypotheses a lot faster than nature can answer them, and every one still needs an experiment that costs lab or telescope time, sometimes for years. So somebody has to pick which hypotheses get that time, and decide which results are real.

With modern LLMs there's also a huge opportunity to go through archival data, and not just in astronomy. Telescopes, particle detectors and sequencers have years of data that only a few people ever looked through.

But if you search enough archival data for enough things, you find signals in pure noise that look real. Physicists call it the look-elsewhere effect, and it's why particle physics wants five sigma before anyone says discovery. An AI testing a million hypotheses a day runs straight into this without trying to cheat. Even at 5σ5\sigma5σ, a million tests a day on pure noise gives you about a hundred fake discoveries a year.

So an AI needs the same rules good scientists follow. Say what you're looking for before you search, and test whatever you find on data you held back.

So far AI only does the digital part of science. I think it'll slowly take over the whole process, including the parts that happen in the real world. It'll run fully automated biology, chemistry and physics labs, and humanoid robots will build the experiment setups. Once it's there, I think progress speeds up exponentially.

If verification is the bottleneck, we should spend the next few years building more ways to ask nature. That means enough automated labs and telescopes for AI to run its experiments on.

Google already has a small one running. Their agents plan an experiment, try it on a simulated copy of the lab first, and then run it on a lab robot. They used it to design short pieces of DNA that stick to a target protein. Humans still approve what gets made and watch the robot.

I'm a techno-optimist. I want humanity to understand and control a lot more of the physical world than we do today, and so far nature is winning. So the plan is to give gradient descent laboratory equipment and tell it to figure out the universe, don't reward hack. There's no conceivable way this ends in a funny incident.

But an agent never touches nature directly. It sees instruments, proxies and a reward function, and all three can be gamed. An RL agent does whatever gets rewarded, so it'll game the measurement if that's easier than finding something real. Google lists this as an open problem, keeping agents on real results "rather than experimental shortcuts". In one run the agents designed DNA that sticks to almost everything, which passes a test that only asks whether it sticks to the target.

Reward hacks in a real lab will be annoying, expensive and sometimes dangerous. The first thing to go wrong is probably human safety. In the early days all of this is still human-assisted. Somebody has to load the samples and build the setup, and that somebody can still say no. Once robots do all the work, nobody is standing there to say no.

The wet lab is the part that scares me. An automated lab could make a bioweapon by accident. I still think we should build them, but only with every safeguard we have.

I'd want containment, checking every DNA design before it's made, and a human approving every run, most of which Google's setup already does. The safest place to start is astronomy, where the worst outcome is a pulsar that isn't there.

My guess for the first funny incident is an automated lab that finds a new particle, and it's the microwave. Or a new microbe, and it's whoever handled the samples. Both have happened to humans. The Parkes telescope picked up mystery radio bursts for years until astronomers traced them to the kitchen microwave. German police hunted a woman whose DNA was at about 40 crime scenes. It came from a worker at the cotton swab factory.

Both got caught because somebody understood the setup well enough to get suspicious.

Nature is the final verifier, but only through an instrument somebody understands.