
The free upgrade that raised your bill
Four hidden factors can move your bill as much as the price per token, or more. On our 191 requests, a 33% price cut came back as a 4% bigger invoice.
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Four hidden factors can move your bill as much as the price per token, or more. On our 191 requests, a 33% price cut came back as a 4% bigger invoice.
Somewhere in your company there's a line on last month's invoice that is higher than the month before, and nobody can quite say why. The product didn't change, the number of users didn't change, and the only thing anyone did was move to the new version of the model, because it was better and its list price was a third lower. The bill went up anyway.
This spring a group of researchers at Stanford, Berkeley, Carnegie Mellon and Microsoft put a number on a related surprise. Across the model pairs they tested, the model with the lower list price cost more to run 32% of the time, and in the worst case by a factor of 28.
That finding was widely shared, and it deserved to be, but it's a story about choosing the wrong model. What we wanted to understand is the less visible case, the one that happens when you choose nothing at all: same feature, same prompt, same vendor, and the next version of the model you were already paying for.
Opinions were easy to find, so we decided to measure instead. For this post we put together a workload that imitates a feature many products have, a reading assistant that takes a paragraph of a technical document and does ordinary things with it: a one-sentence summary, a two-sentence explanation for someone who hasn't read the paper, a question with a numeric answer, an extraction into JSON, a translation into Spanish.
The document is "Attention Is All You Need", partly for the symbolism and mostly because its source is public, so anyone can rebuild our 191 requests and repeat what we did.
We sent those requests to seven versions of two vendors' models, between three and five times each, and priced every billed token at the vendor's own list price. What follows is what we found.
A model doesn't charge you per word. It charges you for the pieces it chops your text into, which the industry calls tokens.
The chopping isn't done word by word. Before training, each model is given a fixed vocabulary of tokens, somewhere in the tens of thousands of them, chosen so that common words like "the" or "model" are a single token while a rarer word like "convolutions" is assembled from two or three. The part of the model that does this chopping is the tokenizer, and a new version of a model often comes with a new vocabulary, so the same text breaks into a different number of tokens.
Take the sentence that started all of this, the one in which the paper introduces its invention: "We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely." Sonnet 4.6 cuts it into 28 tokens. Sonnet 5, the version that replaced it, cuts the very same sentence into 48.
The same sentence, chopped by two versions of the same model
“We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.”
Same sentence, same model family, 71% more tokens.
Same text, by what you send. Our whole 191-request workload: 1.37x.
The same sentence, chopped by two versions of the same model
“We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.”
Same sentence, same model family, 71% more tokens.
Same text, by what you send. Our whole 191-request workload: 1.37x.
Nothing about the sentence changed, and nothing about its meaning. The tokenizer changed, and since you pay per token, the same sentence now takes 71% more of them to read.
How much more depends on what you send, as the bars on the right show: prose and Spanish suffer most, code and JSON least, and across our whole workload it came to 1.37x. It isn't a Sonnet quirk either; Opus 4.6 and Opus 4.7, two versions that list at an identical price, differ by 1.40x on the same text.
So there is an exchange rate between versions, and it isn't printed anywhere on the price sheet.
In June, Anthropic cut Sonnet's price by a third. Anthropic listed Sonnet 4.6 at $3 per million input tokens and $15 per million output tokens, and Sonnet 5 arrived at $2 and $10. If you assumed your bill fell by a third after the switch, you probably haven't checked it.
Sonnet 4.6 to Sonnet 5: the price sheet and the invoice, same 191 requests
Thinking off on both. With both models on their defaults the bill rose 6%.
Sonnet 4.6 to Sonnet 5: the price sheet and the invoice, same 191 requests
Thinking off on both. With both models on their defaults the bill rose 6%.
On our 191 requests, with thinking switched off on both versions, the list price fell 33% and the bill rose 4%. With both models left on their defaults, the setting you get unless you change it, it rose 6%.
Part of this is the exchange rate from the previous section, since the input grew 1.37x. The larger part is what came back. The output grew 1.66x, partly because the finer chopping applies to what the model writes as well as to what it reads, and partly because Sonnet 5 says more.
Given the same paragraph and the same instruction, a one-sentence summary went from 48 billed tokens to 90, and a two-sentence explanation from 82 to 160. Vendors bill a token written at five times the rate of a token read, $10 against $2, so the longer answer is where most of the money went: output was about two thirds of the bill before the upgrade and over 70% after it.
Sonnet 5 is the better model. It answered every one of our fact questions, though so did its predecessor, so we can't show you a quality gain here, only that nothing was lost. What we can show is that a 33% price cut, applied to a feature nobody touched, produced a bigger invoice.
Then we looked at the harder case, the same price twice. Three pairs qualify. Sonnet 5 to Sonnet 5.5 is $2 and $10 both times, and so is GPT-6 Sol to GPT-6.1 Sol; we sent the 191 requests to all four, at their defaults and at their cheapest settings.
Opus 4.6 to Opus 4.7 is $5 and $25 both times, an older upgrade that we ran with thinking off, so that only the chopping and the length of the answer could move. Then we separated each bill into its causes, one at a time.
Four ways a bill moves while the price per token stands still
Opus 4.6 to 4.7, identical $5 / $25, thinking off
Sonnet 5 to 5.5, identical $2 / $10, both on defaults
GPT-6 Sol at none to GPT-6.1 Sol at low, identical $2 / $10
One measurement each, between two versions at the same price. They do not add up.
Four ways a bill moves while the price per token stands still
Opus 4.6 to 4.7, identical $5 / $25, thinking off
Sonnet 5 to 5.5, identical $2 / $10, both on defaults
GPT-6 Sol at none to GPT-6.1 Sol at low, identical $2 / $10
One measurement each, between two versions at the same price. They do not add up.
The chopping. This is the leak people have heard of, and in these two pairs it happens to be zero, since Sonnet 5 and 5.5 share a tokenizer and so do the two GPT-6 Sols. It has happened at an unchanged price, though: Opus 4.6 and Opus 4.7 both list at $5 and $25, and the same 191 requests are 40% more input tokens on the newer one. The other three leaks are the ones that surprise people.
The longer answer. A new version tends to write more for the same question, though not always. Opus 4.7, with thinking off and at the same price as Opus 4.6, wrote 51% more output tokens for the same 191 requests, and between the two leaks its bill came out 47% higher. On their defaults, where neither version thinks, the gap narrows to 35%, because Opus 4.6 writes more on its defaults.
GPT-6.1 Sol at low effort wrote 12% more than GPT-6 Sol at low effort, and cost 8% more for it. Sonnet 5.5 at its cheapest setting, in contrast, produced exactly the same bill as Sonnet 5 at its cheapest, by writing slightly shorter answers on some tasks and slightly longer ones on others.
The default that thinks. Left on its defaults, Sonnet 5.5 decided to reason before answering on 12% of our requests, including questions whose entire answer is a single number. The vendor bills reasoning as output, so the bill rose 7% overall, and on the fact questions, where thinking is pure overhead, it rose 50%. Nothing in the request asked for any of this; the model's idea of a sensible default moved, and the invoice moved with it.
The setting that vanished. GPT-6 Sol offered a reasoning setting called none, and teams tuned their cheapest requests around it. GPT-6.1 Sol doesn't have it. The request a team had carefully tuned yesterday comes back today as an error, and its nearest replacement, low, costs 14% more and takes more than three times as long to answer.
The cheapest request you could send yesterday
The cheapest way to run the feature. Median answer in 1.7 s.
'none' is not supported with this model
The same request, sent to the new version at the same price, comes back as an error.
The lowest setting that still exists. Median answer in 5.0 s.
Same 191 requests, same $2 / $10 list price, same prompt.
The cheapest request you could send yesterday
The cheapest way to run the feature. Median answer in 1.7 s.
'none' is not supported with this model
The same request, sent to the new version at the same price, comes back as an error.
The lowest setting that still exists. Median answer in 5.0 s.
Same 191 requests, same $2 / $10 list price, same prompt.
At least this one fails visibly, with an error. The others don't fail at all; they arrive, weeks later, as a number on an invoice.
We measured each of these on its own, and they don't add up to one tidy number, which is exactly the point. A bill can leak through one of them or through all four at once, and the price sheet, honest as it is, shows none of them.
None of this means upgrades always cost more. On these requests GPT-6.1 Sol on its defaults was 6% cheaper than GPT-6 Sol on its defaults, because the newer version reasons less often. And the same Sonnet upgrade that cost 7% more overall was 12% cheaper on JSON extraction and 50% more expensive on fact questions.
Sonnet 5 to Sonnet 5.5, same price: change in cost per request, by task
Right of zero, Sonnet 5.5 cost more than Sonnet 5; left, less.
Sonnet 5 to Sonnet 5.5, same price: change in cost per request, by task
Right of zero, Sonnet 5.5 cost more than Sonnet 5; left, less.
Switch reasoning off and the ranking changes, as the lower bars show: the explanation task, 8% more expensive on defaults, is 12% cheaper with reasoning off. There is no such thing as what Sonnet 5.5 costs. There is what it costs on your task, at your setting, and that number can go from negative to positive between two tasks of the same feature.
Because the same 191 requests went to both vendors, the runs also allow a comparison we had not set out to make: the two newest models against each other, at the same $2 and $10. Sonnet 5.5 cost 53% to 61% more than GPT-6.1 Sol, and most of that gap is the exchange rate again.
The two newest models at the same $2 / $10, same 191 requests
The two newest models at the same $2 / $10, same 191 requests
GPT-6.1 Sol's lower bill came with higher latency. Compared with GPT-6 Sol, its median response time was 1.5 times longer on defaults and 3.4 times longer at the lowest reasoning setting, which for a feature with a latency budget is a cost of its own. Both facts appear again in the replay table later in the post, measured through three.dev rather than our scripts.
Not the vendor, whose price sheet is perfectly honest and tells you none of this. Not a benchmark, which measures its own questions at a setting you may never use. We saw how far the leaderboard favorite can be from the right choice when we replayed one real task through seven models,
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。