返回资讯中心
外部精选
软件工程
#开发者实时流

Bitter Lessons and Premature Optimizations

Donald Knuth wrote “premature optimization is the root of all evil”

Hacker News 最新hellerve2026/10/17 · 2026/10/1713 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Donald Knuth wrote “premature optimization is the root of all evil” in 1974, in a paper called “Structured Programming with go to Statements”. It’s high on the leaderboard of the most quoted sentences in our field, and I would wager that only a fraction of the people quoting it have read the paragraph it comes from.

Here it is, with the part I think is important but quite often missed:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified.

Fig. 1: Knuth, unabridged-ish.

The argument is about where to spend your effort. Don’t waste your time on code that doesn’t matter, find the code that does, and then optimize the hell out of it. In the very next sentence he explains how you find it. Of course you measure, because “the universal experience of programmers who have been using measurement tools has been that their intuitive guesses fail”1.

That’s not what the sentence turned into, of course. Culturally, “premature optimization” became shorthand for “don’t optimize”, and the word premature becomes irrelevant. Someone suggests making a thing faster, someone else recites the sentence, and the conversation is over. This happens without anyone having established that the optimization was premature, or what premature means (presumably we’ll get to it when the first requests start timing out).

This post is not about that, though. Instead, look at it as a historical precedent, because I think the Bitter Lesson is going through the same thing right now.

In 2019, Rich Sutton published a short essay called “The Bitter Lesson”. It opens like this:

The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.

Fig. 2: Sutton’s thesis, in his own words.

He walks through chess, Go, speech recognition, and computer vision. Each time, researchers built what they knew about the domain into their systems. It helped in the short term, then it plateaued, and eventually it lost to methods that leaned on ever-cheaper computation. His verdict: “The two methods that seem to scale arbitrarily in this way are search and learning.”

It’s a good essay, and I agree with most of it, begrudgingly. But a few things are worth pointing out, because, again, the nuance tends to get lost.

Sutton isn’t saying that algorithms don’t matter. Finding methods that make good use of computation matters a great deal! He also isn’t only talking about training, since search at play time is one of his main examples. And the claim is about how to build methods that keep improving as computation scales, not about every system those methods end up in.

But it’s also a pretty constrained essay that leaves many things unsaid, and people are starting to put words in its mouth already.

The thing I see most often is a slide from general methods to general models to general systems, as if they were the same thing. A general learning method can produce a very narrow model. A general model can be used to train a specialist, for instance by fine-tuning it on a single task, or by using it as a teacher for a model that only ever sees that task. And none of this says anything about whether the hardware it runs on should be general. That would required quite a different argument.

The essay also isn’t a theory of cost. It doesn’t tell you to run a large generative model when a small classifier does the job. It doesn’t tell you how to weigh one training run against the billions of inferences that come after it, or when the cost of keeping a system general starts to exceed what you get out of it, or when engineering effort pays for itself in operating cost. This is hard, bespoke work, and only tangentially related to engineering (though it can be influenced by it).

A method that scales better is not necessarily a system that’s cheaper to run. A general method might be the right way to find a solution, and a specialized system might be the right way to deploy and run it!

I’d argue that the choice between the two is mostly economic. Generality saves you from engineering each solution separately. Specialization can save you recurring cost. That matters more as a workload becomes predictable and its volume grows, because the engineering cost gets amortized. This is true for software, for models, and for physical machines, with very different numbers attached to each, many of which I don’t know, because my expertise is pretty narrow in this domain.

Specialization doesn’t always win, of course. But we should do the calculation and apply diligence instead of just assuming the answer. If anything, general learning methods might make specialization cheaper, because they let us derive the specialist from the generalist instead of building it by hand.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →