
外部精选
软件工程
#开发者实时流Switching LLMs mid-conversation is a ski rental problem
<p>Most work on routing between language models answers one question: which model should handle this
Hacker News 最新kokakiwi2026/11/10 · 7 天前30 分钟阅读
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
<p>Most work on routing between language models answers one question: which model should handle this
request? A classifier scores the request, and if it's an easy one directs it to a cheap model while a hard one goes
to a frontier model. For a chatbot answering one-off questions independently, that is the whole characterization of the problem.</p>
<p>For a coding agent it is only half of it. A coding session is a long conversation in which the agent
resends everything said so far with every request, and LLM providers cache that history so they need
not reprocess it each time. A router that picks the right model for every turn, but ignores the
cache, throws it away at each change of model and pays to rebuild it. The second half of the problem
is therefore <em>when</em> a running session should change model. That is an online problem, decided turn
by turn without knowing the future, between a one-off cost and a stream of small savings, and it has
a classic form: the ski-rental problem.</p>
<p>Edgee's Agent Gateway treats routing as both halves. A <strong>tier map</strong> assigns a model to each level of
difficulty, and a <strong>switching policy</strong> based on ski rental decides when a session actually moves.
Self-Learning Routing learns both from an organization's own traffic. The gateway labels every turn
as it serves it. Once a week the control plane replays the past 30 days of sessions through a copy of
the gateway's algorithm, under every candidate map and several switching settings, and measures each
against the schedule that hindsight would have chosen. It recommends the cheapest combination that
keeps every turn on a capable model, and applies it when a series of guardrails allow.</p>
<h2>Why difficulty is only half the story</h2>
<p>Picking a model once per session is easy, and wasteful: a session that opens with a hard design
question often spends the next hour on edits a far cheaper model could make. The savings come from
moving mid-task. But every move has a price.</p>
<p>On Anthropic, caching a prefix for five minutes costs 1.25 times the normal input price, and reading
it back costs a tenth (<a href="https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching">prompt caching</a>).
A session that changes model loses its warm cache. The new model reads the whole conversation cold
and writes it to its own cache. For a long session, that one rewrite can cost more than many turns of
the price difference between the two models.</p>
<p>So the two halves interact. A tier map that puts a cheap model on easy turns only saves money if
sessions actually move down to it, and stay long enough to recoup the rewrite. Switch on every turn
that looks easy and the cache rewrites eat the savings. Never switch and the cheap model is never
used. Which map is best therefore depends on the switching policy, and the switching policy that is
best depends on the map. Self-Learning Routing learns them together for that reason.</p>
<h2>Half one: difficulty tiers</h2>
<p>At the start of each user turn the gateway asks a classifier one question about the latest message:
how much capability does this task need? The answer comes on a four-point scale, from a trivial edit
to the hardest architectural problem, as a probability-weighted level index from 0 to 3. The gateway
normalizes it to a score in <code>[0, 1]</code> and maps the score to a tier:</p>
<table>
<thead>
<tr>
<th>Score</th>
<th>Tier</th>
<th>What it looks like</th>
</tr>
</thead>
<tbody>
<tr>
<td>≤ 0.2</td>
<td>low</td>
<td>A typo, a rename, formatting</td>
</tr>
<tr>
<td>≤ 0.5</td>
<td>medium</td>
<td>A small function, a straightforward bug</td>
</tr>
<tr>
<td>≤ 0.8</td>
<td>high</td>
<td>A change across several files, a subtle bug</td>
</tr>
<tr>
<td>> 0.8</td>
<td>max</td>
<td>Architecture, security-critical or open-ended work</td>
</tr>
</tbody>
</table>
<p>The requests inside a turn, as the agent calls tools and reads their output, never decide afresh:
they stay on the model the turn began with.</p>
<p>Moving up a tier is a constraint, not a bet. A turn that needs a more capable model gets one, at once
and whatever the switching cost: quality is not traded for money. The only filter is the
classifier's confidence, so that a single noisy label cannot drag a session upmarket:</p>
<table>
<thead>
<tr>
<th>Confidence in the upgrade</th>
<th>Decision</th>
</tr>
</thead>
<tbody>
<tr>
<td>≥ 0.75</td>
<td>Upgrade now</td>
</tr>
<tr>
<td>0.5 to 0.75</td>
<td>Upgrade once two consecutive turns ask for it</td>
</tr>
<tr>
<td>< 0.5</td>
<td>Stay on the current tier</td>
</tr>
</tbody>
</table>
<p>Upgrades sit outside the cost accounting that follows. Everything else, moving to a cheaper model or
to a different model for the same tier, is a cost decision, and that is the second half.</p>
<h2>Half two: switching as a ski-rental problem</h2>
<h3>Online algorithms</h3>
<p>An algorithm is <em>online</em> when it must act on each part of its input as it arrives, without seeing
what comes next. Its <em>offline</em> counterpart is handed the whole input in advance. Caching is the
classic example: a full cache must evict an entry now, and only later requests reveal which entry
was not needed again.</p>
<p>Online algorithms are judged by <em>competitive analysis</em> (Sleator and Tarjan, <em>Amortized efficiency of
list update and paging rules</em>, Communications of the ACM, 1985). An online algorithm is
<em>c-competitive</em> if, on every input, it pays at most <code>c</code> times what the best offline algorithm pays on
that same input. The input may be chosen by an adversary who knows the algorithm, so the ratio is a
promise about the worst case, not the average one.</p>
<p>Routing a coding session is an online problem: the gateway must pick a model for this turn before the
developer has written the next one. Half one barely notices, since a turn's tier depends only on that
turn. Switching does. Whether a move pays depends on the past, which decides where the warm cache is,
and on the future, which decides how long the cheaper model will stay the right one. The gateway
knows only the past.</p>
<h3>Renting and buying</h3>
<p>The simplest online problem with that shape is ski rental. A skier can rent skis for a fee each day
or buy a pair for <code>B</code>, without knowing how many days the season will last. Renting through a long
season costs far more than the skis; buying on the first day wastes the money if the season ends the
next morning. The problem was first analyzed in this form for caches too: multiprocessor caches
deciding whether to keep paying to update a shared block or to drop it (Karlin, Manasse, Rudolph and
Sleator, <em>Competitive snoopy caching</em>, Algorithmica, 1988).</p>
<h3>Why break-even is 2-competitive</h3>
<p>Let <code>S</code> be what renting would cost over the whole season. The offline optimum knows <code>S</code>, so it pays
<code>min(S, B)</code>: it rents through a short season and buys on the first day of a long one. The break-even
rule rents until the rent paid reaches <code>B</code>, then buys. (Rent is treated as continuous, a fair
approximation when a day's rent is small next to the price of skis, as a turn's saving is next to a
cache rewrite.)</p>
<ul>
<li>If the season ends before the rent paid reaches <code>B</code>, the rule has rented throughout, as the optimum
did, and pays <code>S</code>.</li>
<li>Otherwise it pays <code>B</code> in rent, then <code>B</code> for the skis: <code>2B</code>, against the optimum's <code>B</code>.</li>
</ul>
<p>Either way it pays at most twice the optimum. Its worst case is the one an adversary would pick: the
season ends the moment the skis are bought.</p>
<p>The same adversary shows that no deterministic rule does better. Any such rule amounts to buying once
the rent paid reaches <code>x·B</code>, for some <code>x</code>. End the season just after the purchase: the rule pays
<code>(1 + x)·B</code> and the optimum <code>min(x, 1)·B</code>, a ratio of</p>
<pre class="hljs"><code class="hljs text">1 + max(x, 1/x)
</code></pre>
<p>That is also the worst any season can do to the rule, so it is the rule's exact competitive ratio. It
is smallest at <code>x = 1</code>, where it equals 2. Buying early loses to a short season, buying late loses to
a long one, and break-even balances the two regrets. Randomizing the moment of purchase lowers the
bound to <code>e/(e−1) ≈ 1.58</code> (Karlin, Manasse, McGeoch and Owicki, <em>Competitive randomized algorithms
for nonuniform problems</em>, Algorithmica, 1994). The gateway stays deterministic: the same session
always gets the same decisions, which is what lets the replay re-run them.</p>
<h3>Our problem as ski rental</h3>
<p>When a cheaper model becomes the right one for a session's turns, the gateway faces the skier's
choice:</p>
<table>
<thead>
<tr>
<th>Ski rental</th>
<th>A coding session</th>
</tr>
</thead>
<tbody>
<tr>
<td>A day</td>
<td>A user turn</td>
</tr>
<tr>
<td>Renting</td>
<td>Staying on the current model, and paying each turn the difference with what the cheaper model would cost if its cache were warm too</td>
</tr>
<tr>
<td>Buying</td>
<td>Switching, and paying once for the cheaper model to read the whole conversation cold and cache it</td>
</tr>
<tr>
<td>The season</td>
<td>The run of turns for which the cheaper model remains the right one</td>
</tr>
<tr>
<td>The adversary</td>
<td>The developer, whose next request nobody can predict</td>
</tr>
</tbody>
</table>
<p>The fit is close but not exact, and each difference shaped the design:</p>
<ul>
<li><strong>The price of skis rises.</strong> A switch rewrites the whole conversation, which usually grows with
every turn, so the longer the gateway waits, the dearer buying gets. The rent moves too, with the
length of the answers. The gateway re-prices both on every turn. The proof above assumes fixed
prices, so in a real session it is a guide rather than a theorem; the replay measures what the
rule actually leaves on the table.</li>
<li><strong>Seasons repeat.</strong> A session holds many stretches, not one. When the candidate changes, because
the turn's tier or the Strategy's map changed, a new instance begins, and the evidence gathered
against the old candidate stops counting.</li>
<li><strong>The skis can be taken away.</strong> A hard turn forces an upgrade whatever it costs, which ends the
stretch; coming back down means buying again. A threshold above break-even guards against exactly
this.</li>
<li><strong>Sometimes the skis are free.</strong> When the cache has expired or the conversation has just been
compacted, the current model must rewrite the conversation anyway, so switching costs nothing
extra.</li>
</ul>
<h3>The cost model</h3>
<p>For each turn the gateway estimates four token counts: <code>P</code>, the conversation prefix already warm on
the current model <code>p</code>; <code>N</code>, the tokens new this turn; <code>O</code> and <code>R</code>, the expected output and reasoning
tokens, taken from running averages of the session's own past turns. With a model's rates for cache
writes <code>w</code>, cache reads <code>h</code>, output <code>o</code> and reasoning <code>ρ</code>, and <code>c</code> the cheaper candidate:</p>
<pre class="hljs"><code class="hljs text">cold(m) = (P + N)·w(m) + O·o(m) + R·ρ(m) # nothing cached: the whole prefix is written
warm(m) = P·h(m) + N·w(m) + O·o(m) + R·ρ(m) # the prefix is a cache hit
buy B = max(0, cold(c) − warm(p)) # one-off premium of switching this turn rent r = max(0, warm(p) − warm(c)) # saving forgone per turn by staying </code></pre>
<h3>The break-even rule in the gateway</h3> <p>The gateway keeps a running total <code>A</code> of the rent paid against the current candidate, and buys when it reaches a multiple of the buy price:</p> <pre class="hljs"><code class="hljs text">A ← A + r on every turn the candidate stays the same switch when A ≥ λ·B正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。