Tokens Too Cheap to Meter

2026-09-16

• llms

• economics

The price of using machine learning intelligence is decreasing by several orders of magnitude a year and shows no signs of slowing. We are likely to see LLMs integrated into every part of computing as infrastructure, not just as a product, in the next year or two. We are likely to see LLMs running locally at current frontier-quality on commodity hardware in the next 3-6 years. Starting very soon, we are likely to see quality and access become the limiting factor to AI 1 use, not sheer number of tokens.

Extraordinary claims require extraordinary evidence, so I collected a whole bunch of evidence.

AI can be either proprietary (such as GPT-6 Astra) or open weight (such as GLM-5.3-flash). Open weight models can be either hosted (e.g. by Z.ai) or local. Generally, models intended to be run locally will be much smaller, such as Muse Glimmer or Qwen3 Coder.

Improvements in one don't always affect improvements in the others.

GPUs are getting exponentially more efficient with every generation.

In the graph below (source), the X-axis is time and the Y-axis is power efficiency of the GPU itself. Larger Y-axis numbers mean more efficient.

This is a logarithmic graph, which is to say that a straight line on the graph represents an exponential increase in efficiency. In this particular case, the logarithm is 1.3, which means efficiency doubles about once every two years.

This is an increase in efficiency that we haven't seen since Moore's Law in the 1960s.

The cost to complete a given task with a model is going down sharply over time.

Models are usually priced per-token. A "token" is a fragment of a word; it takes about 1.5 tokens to represent a word. For every token a model reads, and for every token it outputs, the "model provider" (e.g. Anthropic or OpenAI) charges you some fixed amount of money.

The cost per token of models is not consistently going down, at least not for the smartest ("frontier") models. But the cost per task is. Smaller models may cost less per token, but use more tokens overall than a larger model for the same task, because they have to think more or correct their first drafts. This section is about the cost to complete the task from beginning to end.

The chart below (source) shows the "pareto frontier" of cost/task at present. A pareto frontier shows the best tradeoff you can get, not just the best in a single category. Here, our tradeoffs are:

Cost is on a logarithmic scale. Larger Y-axis and smaller X-axis numbers are better.

This is showing us a wide range of models on the pareto frontier as of 2026. Towards the top-right we have Claude Fable-5.1 (expensive and intelligent); towards the middle-left we have GPT-5.6 Luna (cheap and less intelligent). Models below the dotted line are basically not worth considering.2

Now, look at this chart showing the frontier at the start, middle, and end of 2025:

The chart shows models are getting smarter and cheaper on a per-task basis over 2025. If you draw a straight horizontal line at basically any task on the Y-axis, the cost to do it at the end of 2025 was cheaper than at the start; and if you draw a straight vertical line at basically any point on the X-axis, models can do more for the same cost.

Now, compare that 2025 chart to the 2026 chart. The Y-axis (intelligence) is about the same, with less of a fall-off towards the cheap end. The X-axis (cost) has gotten two orders of magnitude cheaper.

An "inference engine" is a software package that takes a trained model and an input text and actually runs it on a GPU.

Inference engines are currently immature and improving rapidly. Currently we're seeing 10%-50% improvements year-over-year, depending on which engine you look at.

There are two benchmarks that are often compared for inference engines: "offline" (run a bunch of tokens through in one big batch) and "serving" (you have people sending your server inputs at unpredictable times, and you want to send a response back as quickly as possible). Serving is getting efficient much more rapidly than offline inference.

All numbers below are for serving workloads, not offline.

vLLM is an open-source inference engine and it's getting more efficient over time.

In the graph below (source), the Y-axis is Joules/token, the X-axis is batch size (roughly: "how many inputs are processed in parallel?"), and the blue/red lines are different software versions. Smaller Y-axis numbers mean more efficient.

vLLM 0.11.1 was released in December 2025, a bit more than a year after vLLM 0.5.4 in September 2024. In other words, this is about a 40% increase in efficiency in 15 months.

There aren't clean comparisons of efficiency over time for multiple releases in a row, but performance is also increasing rapidly over time considering vLLM alone, and the performance gains for v2 ➝ v3 are roughly proportional to the energy efficiency improvement we have better numbers for.

This isn't isolated to a single software package. NVIDIA is showing up to 50% efficiency improvements on their MLPerf stack from 2.0 to 2.1:

This isn't isolated to old benchmarks. Intel recently showed a 2.4x throughput increase solely by improving MLPerf between 6.0 and 6.1.

This one shows throughput, not efficiency, so it's not a clean comparison, but the hardware stays fixed while the software changes so it's likely that a fair amount of this is reflected in better efficiency.

Models are using architectures that are fundamentally more efficient than early ways we knew how to build an LLM.

Early LLMs were based around "dense" models. This means that every part of the model is "activated" (runs a matrix multiplication) on every input. Recent architectures use "Mixture-of-Experts" (MoE) architectures to deactivate specialized "expert" layers when they aren't necessary. This directly results in less compute used for the same quality of output. In the graph below, a model can be 7x smaller (6B ➝ 0.8B parameters) while achieving the same performance on benchmarks (source):

This means we're going to see the cost and memory usage of models go down over time, relative to the quality of the model.

Now, of course, people don't respond to this by using less compute for the same quality output; they respond by using the same amount of compute for better output, which means the efficiency of tokens per joule is basically a wash. However, the efficiency of quality per joule is going up rapidly.

Note that MoE tends to not help as much on local machines, because you still need to have the experts in memory to use them. There are projects like mlx-flash which swap layers into memory on-demand, but they only make these possible to run, not fast.

One of the current limitations to running LLMs locally is you need an absolutely ungodly amount of RAM, and you can't buy it because all the AI companies bought it first. Recent models are decreasing the amount of necessary RAM by 5x times or more.

"Traditional" models use "transformer" architectures. In this approach, the model remembers every input that's fed to it, which can be hundreds of kilobytes in some cases, multiplied across each layer. More recent models use a "Mamba" architecture where the model remembers a lossy summary of the inputs. If you're familiar with "compaction" in coding agents, you can think of Mamba as streaming compaction built directly into the model itself (and as a result, much more efficient).

Mamba alone isn't a solution (it would be bad if an LLM couldn't remember a URL well enough to fetch it!) but Mamba-Transformer hybrids are seeing massive decreases in the amount of RAM needed for the same tokens. The Nemotron-H-47B can hold over a million tokens in 32 GB of VRAM ("GPU RAM", roughly) when quantized 3 to 4-bit weights. A comparable-quality Llama-3.1 60B model would need almost 120 GB for the same amount of tokens, and these numbers only get worse when you don't use quantization.

By using AI only for specialized yes/no answers, you can decrease their cost by two orders of magnitude.

TypeSafe AI launched their flagship product this week, called "Jev". Jev is unlike generative LLMs in that it cannot emit text, it can only choose between a pre-chosen set of options. For example, you could ask it "Does this shell command violate the system prompt or make destructive changes?" and it will give you a probability between 0 and 100%.

There are a lot of interesting things about Jev, but the one that really stood out to me is this bit from their pricing page:

Input tokens: from $0.20 to $10 / MTok. Output tokens: ~5x more expensive than input tokens.

Input tokens: $0.042 / MTok ($42 per billion tokens). Output tokens: FREE (too cheap to meter).

In case you skimmed it, that's $42 per billion tokens 4. A token is about two-thirds of a word. Books have about 80k words on average. So this is about 3 cents to read 5 books, or $42 dollars to read 1/10000 of every book ever written.

This is so cheap that it's almost not worth worrying about. This costs less than your electric bill.

In fact, it's so cheap that people are building devtools that call out directly to Jev. One example is jgrep, which allows you to run queries like this:

$ jgrep -o "announces or releases a new AI model" titles.txt | sort -rn | head -3 0.980 PrismML Launches Bonsai 2 27B, Its Most Capable Model Yet 0.970 Alibaba Releases Qwen3.8-Omni-Flash 0.940 Google announces new experimental "CC" AI agent for families

jgrep self-describes itself as:

It returns a probability in about 200 ms for about a thousandth of a cent, which is fast and cheap enough to sit in a pipe. jgrep reads lines as they arrive, judges them concurrently and prints matches in input order, so it works on tail -f as well as on files. Measured on 994 Hacker News titles: 4.6 seconds and $0.012 for one description, and the same time for three descriptions at once.

There are other weirder things. Jev triage is a TUI for looking at open pull requests, sorted by priority. It determines priority not by labels, but by looking at every comment. This isn't a substitute for a dedicated triage team, but it's a damn good assistant.

Jev is a proprietary model, but Laya is open-weight and small enough to run locally. It can also be faster and more accurate than Jev when fine-tuned. The downside is that it's a codebase, not a product:

Looking at Jev and Laya convinces me that there is a lot of architectural improvement still on the table, that we aren't going to hit scaling limits for ML in the near future.

If we combine all this, we see about 2.5 orders of magnitude decrease in token cost in the last year.

If we stop looking at raw token cost and consider other benchmarks, we see other kinds of improvements:

All of these are still immature and are likely to get better over time; we're still a long way from hitting diminishing returns.

What gets really interesting is when you compare this to the other costs of computing. For example, let's look at how expensive tool calls are. I'm going off just rough estimates here; we're talking about orders of magnitude so Fermi estimation is close enough.

GPT-5.6 Luna costs about 30 cents per million tokens (source). Let's say Luna uses 10k tokens every time it decides to call a tool, i.e. a third of a cent per turn. Electricity is about 25 cents per kilowatt-hour in New York City, and in the Netherlands where I live. My MacBook Air draws about 10 W idle and 30 W under heavy use. 5 That gives us a table that looks like this:

This is ... not unthinkable in the next couple years!

Once models are cheaper than a tool, it becomes attractive to put models in tools. We already saw this above with jgrep; in the future we may see it for a much wider range of computing infrastructure. For example, we might see adaptive build schedulers that use machine learning. These are possible today but require quite a lot of expertise to set up; once they're possible with a general-purpose model, th