返回资讯中心
外部精选
软件工程
#开发者实时流

Switching LLMs mid-conversation is a ski rental problem

<p>Most work on routing between language models answers one question: which model should handle this

Hacker News 最新kokakiwi2026/11/10 · 7 天前30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

<p>Most work on routing between language models answers one question: which model should handle this request? A classifier scores the request, and if it's an easy one directs it to a cheap model while a hard one goes to a frontier model. For a chatbot answering one-off questions independently, that is the whole characterization of the problem.</p> <p>For a coding agent it is only half of it. A coding session is a long conversation in which the agent resends everything said so far with every request, and LLM providers cache that history so they need not reprocess it each time. A router that picks the right model for every turn, but ignores the cache, throws it away at each change of model and pays to rebuild it. The second half of the problem is therefore <em>when</em> a running session should change model. That is an online problem, decided turn by turn without knowing the future, between a one-off cost and a stream of small savings, and it has a classic form: the ski-rental problem.</p> <p>Edgee's Agent Gateway treats routing as both halves. A <strong>tier map</strong> assigns a model to each level of difficulty, and a <strong>switching policy</strong> based on ski rental decides when a session actually moves. Self-Learning Routing learns both from an organization's own traffic. The gateway labels every turn as it serves it. Once a week the control plane replays the past 30 days of sessions through a copy of the gateway's algorithm, under every candidate map and several switching settings, and measures each against the schedule that hindsight would have chosen. It recommends the cheapest combination that keeps every turn on a capable model, and applies it when a series of guardrails allow.</p> <h2>Why difficulty is only half the story</h2> <p>Picking a model once per session is easy, and wasteful: a session that opens with a hard design question often spends the next hour on edits a far cheaper model could make. The savings come from moving mid-task. But every move has a price.</p> <p>On Anthropic, caching a prefix for five minutes costs 1.25 times the normal input price, and reading it back costs a tenth (<a href="https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching">prompt caching</a>). A session that changes model loses its warm cache. The new model reads the whole conversation cold and writes it to its own cache. For a long session, that one rewrite can cost more than many turns of the price difference between the two models.</p> <p>So the two halves interact. A tier map that puts a cheap model on easy turns only saves money if sessions actually move down to it, and stay long enough to recoup the rewrite. Switch on every turn that looks easy and the cache rewrites eat the savings. Never switch and the cheap model is never used. Which map is best therefore depends on the switching policy, and the switching policy that is best depends on the map. Self-Learning Routing learns them together for that reason.</p> <h2>Half one: difficulty tiers</h2> <p>At the start of each user turn the gateway asks a classifier one question about the latest message: how much capability does this task need? The answer comes on a four-point scale, from a trivial edit to the hardest architectural problem, as a probability-weighted level index from 0 to 3. The gateway normalizes it to a score in <code>[0, 1]</code> and maps the score to a tier:</p> <table> <thead> <tr> <th>Score</th> <th>Tier</th> <th>What it looks like</th> </tr> </thead> <tbody> <tr> <td>≤ 0.2</td> <td>low</td> <td>A typo, a rename, formatting</td> </tr> <tr> <td>≤ 0.5</td> <td>medium</td> <td>A small function, a straightforward bug</td> </tr> <tr> <td>≤ 0.8</td> <td>high</td> <td>A change across several files, a subtle bug</td> </tr> <tr> <td>&gt; 0.8</td> <td>max</td> <td>Architecture, security-critical or open-ended work</td> </tr> </tbody> </table> <p>The requests inside a turn, as the agent calls tools and reads their output, never decide afresh: they stay on the model the turn began with.</p> <p>Moving up a tier is a constraint, not a bet. A turn that needs a more capable model gets one, at once and whatever the switching cost: quality is not traded for money. The only filter is the classifier's confidence, so that a single noisy label cannot drag a session upmarket:</p> <table> <thead> <tr> <th>Confidence in the upgrade</th> <th>Decision</th> </tr> </thead> <tbody> <tr> <td>≥ 0.75</td> <td>Upgrade now</td> </tr> <tr> <td>0.5 to 0.75</td> <td>Upgrade once two consecutive turns ask for it</td> </tr> <tr> <td>&lt; 0.5</td> <td>Stay on the current tier</td> </tr> </tbody> </table> <p>Upgrades sit outside the cost accounting that follows. Everything else, moving to a cheaper model or to a different model for the same tier, is a cost decision, and that is the second half.</p> <h2>Half two: switching as a ski-rental problem</h2> <h3>Online algorithms</h3> <p>An algorithm is <em>online</em> when it must act on each part of its input as it arrives, without seeing what comes next. Its <em>offline</em> counterpart is handed the whole input in advance. Caching is the classic example: a full cache must evict an entry now, and only later requests reveal which entry was not needed again.</p> <p>Online algorithms are judged by <em>competitive analysis</em> (Sleator and Tarjan, <em>Amortized efficiency of list update and paging rules</em>, Communications of the ACM, 1985). An online algorithm is <em>c-competitive</em> if, on every input, it pays at most <code>c</code> times what the best offline algorithm pays on that same input. The input may be chosen by an adversary who knows the algorithm, so the ratio is a promise about the worst case, not the average one.</p> <p>Routing a coding session is an online problem: the gateway must pick a model for this turn before the developer has written the next one. Half one barely notices, since a turn's tier depends only on that turn. Switching does. Whether a move pays depends on the past, which decides where the warm cache is, and on the future, which decides how long the cheaper model will stay the right one. The gateway knows only the past.</p> <h3>Renting and buying</h3> <p>The simplest online problem with that shape is ski rental. A skier can rent skis for a fee each day or buy a pair for <code>B</code>, without knowing how many days the season will last. Renting through a long season costs far more than the skis; buying on the first day wastes the money if the season ends the next morning. The problem was first analyzed in this form for caches too: multiprocessor caches deciding whether to keep paying to update a shared block or to drop it (Karlin, Manasse, Rudolph and Sleator, <em>Competitive snoopy caching</em>, Algorithmica, 1988).</p> <h3>Why break-even is 2-competitive</h3> <p>Let <code>S</code> be what renting would cost over the whole season. The offline optimum knows <code>S</code>, so it pays <code>min(S, B)</code>: it rents through a short season and buys on the first day of a long one. The break-even rule rents until the rent paid reaches <code>B</code>, then buys. (Rent is treated as continuous, a fair approximation when a day's rent is small next to the price of skis, as a turn's saving is next to a cache rewrite.)</p> <ul> <li>If the season ends before the rent paid reaches <code>B</code>, the rule has rented throughout, as the optimum did, and pays <code>S</code>.</li> <li>Otherwise it pays <code>B</code> in rent, then <code>B</code> for the skis: <code>2B</code>, against the optimum's <code>B</code>.</li> </ul> <p>Either way it pays at most twice the optimum. Its worst case is the one an adversary would pick: the season ends the moment the skis are bought.</p> <p>The same adversary shows that no deterministic rule does better. Any such rule amounts to buying once the rent paid reaches <code>x·B</code>, for some <code>x</code>. End the season just after the purchase: the rule pays <code>(1 + x)·B</code> and the optimum <code>min(x, 1)·B</code>, a ratio of</p> <pre class="hljs"><code class="hljs text">1 + max(x, 1/x) </code></pre> <p>That is also the worst any season can do to the rule, so it is the rule's exact competitive ratio. It is smallest at <code>x = 1</code>, where it equals 2. Buying early loses to a short season, buying late loses to a long one, and break-even balances the two regrets. Randomizing the moment of purchase lowers the bound to <code>e/(e−1) ≈ 1.58</code> (Karlin, Manasse, McGeoch and Owicki, <em>Competitive randomized algorithms for nonuniform problems</em>, Algorithmica, 1994). The gateway stays deterministic: the same session always gets the same decisions, which is what lets the replay re-run them.</p> <h3>Our problem as ski rental</h3> <p>When a cheaper model becomes the right one for a session's turns, the gateway faces the skier's choice:</p> <table> <thead> <tr> <th>Ski rental</th> <th>A coding session</th> </tr> </thead> <tbody> <tr> <td>A day</td> <td>A user turn</td> </tr> <tr> <td>Renting</td> <td>Staying on the current model, and paying each turn the difference with what the cheaper model would cost if its cache were warm too</td> </tr> <tr> <td>Buying</td> <td>Switching, and paying once for the cheaper model to read the whole conversation cold and cache it</td> </tr> <tr> <td>The season</td> <td>The run of turns for which the cheaper model remains the right one</td> </tr> <tr> <td>The adversary</td> <td>The developer, whose next request nobody can predict</td> </tr> </tbody> </table> <p>The fit is close but not exact, and each difference shaped the design:</p> <ul> <li><strong>The price of skis rises.</strong> A switch rewrites the whole conversation, which usually grows with every turn, so the longer the gateway waits, the dearer buying gets. The rent moves too, with the length of the answers. The gateway re-prices both on every turn. The proof above assumes fixed prices, so in a real session it is a guide rather than a theorem; the replay measures what the rule actually leaves on the table.</li> <li><strong>Seasons repeat.</strong> A session holds many stretches, not one. When the candidate changes, because the turn's tier or the Strategy's map changed, a new instance begins, and the evidence gathered against the old candidate stops counting.</li> <li><strong>The skis can be taken away.</strong> A hard turn forces an upgrade whatever it costs, which ends the stretch; coming back down means buying again. A threshold above break-even guards against exactly this.</li> <li><strong>Sometimes the skis are free.</strong> When the cache has expired or the conversation has just been compacted, the current model must rewrite the conversation anyway, so switching costs nothing extra.</li> </ul> <h3>The cost model</h3> <p>For each turn the gateway estimates four token counts: <code>P</code>, the conversation prefix already warm on the current model <code>p</code>; <code>N</code>, the tokens new this turn; <code>O</code> and <code>R</code>, the expected output and reasoning tokens, taken from running averages of the session's own past turns. With a model's rates for cache writes <code>w</code>, cache reads <code>h</code>, output <code>o</code> and reasoning <code>ρ</code>, and <code>c</code> the cheaper candidate:</p> <pre class="hljs"><code class="hljs text">cold(m) = (P + N)·w(m) + O·o(m) + R·ρ(m) # nothing cached: the whole prefix is written warm(m) = P·h(m) + N·w(m) + O·o(m) + R·ρ(m) # the prefix is a cache hit

buy B = max(0, cold(c) − warm(p)) # one-off premium of switching this turn rent r = max(0, warm(p) − warm(c)) # saving forgone per turn by staying </code></pre>

<h3>The break-even rule in the gateway</h3> <p>The gateway keeps a running total <code>A</code> of the rent paid against the current candidate, and buys when it reaches a multiple of the buy price:</p> <pre class="hljs"><code class="hljs text">A ← A + r on every turn the candidate stays the same switch when A ≥ λ·B

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →