An AI service can have a sound technical reason to favor a conversation that is already underway: it has done much of the work needed to read that conversation. Serving another turn can preserve that advantage. Starting someone else’s task may mean making room for context the machines have never processed.
That is an uncomfortable place for a queue to acquire its priorities. If I am waiting to begin, “the other conversation has already asked more questions” does not sound like a reason to keep me waiting. If I am midway through a difficult task, continuity sounds rather more reasonable. Recent serving designs make this conflict concrete, down to an explicit choice between favoring established conversations and giving less-served work a turn.
The first request chooses more than a machine
The relevant memory is a KV cache: intermediate model state that lets a later request reuse computation for an identical opening sequence of tokens. In a conversation, the next request can include the earlier exchange plus new material. vLLM’s documentation explains how prefix caching avoids processing that shared history again. The saving applies to reading the prompt; generating the new answer still takes work. A long conversation is therefore a source of both additional work and reusable computation.
In SMetric, published July 9, Jiahao Wang and colleagues study production traces from two clusters serving coding agents. They describe how common opening instructions attract new sessions to the same cache-warm instances. Follow-up requests then return for the context accumulated there. A decision that looks economical for one request can concentrate entire conversations on a few machines.
Their scheduler treats a session’s first request differently, placing it for load balance and favoring local reuse on later turns, with exceptions for overload or eviction. Shared cache storage helps another instance retrieve existing state. It does not make movement free: storage capacity and transfer bandwidth can become constraints. The reported performance experiments replay a production trace with adjustments, rather than measuring new customer outcomes. The scheduler balances a new conversation before its history gives the router a reason to keep sending it back.
There is also a less session-specific answer. In their August 17 account of llm-d’s routing, Kaushik Mitra and colleagues describe retaining cache affinity until a calibrated load limit is crossed, then choosing by load. The signal depends on what is congested: prompt processing and answer generation need different treatment. This is a credible escape from a crowded machine, not an argument for abandoning reuse. Their constructed workload tests do not establish one threshold that every service should adopt.
These approaches weaken the idea that a conversation must stay wherever it began. They also expose a separate decision. Choosing a suitable machine is useful while there is somewhere suitable to go. When the pool is saturated, a service still has to choose which waiting work gets access next.
Continuing is not the same as finishing
llm-d separates that admission decision from backend routing. Its June 24 release describes holding requests in queues before committing them to overloaded model servers, with policies for priority and fairness between flows. This leaves room to choose among waiting conversations, rather than letting an earlier routing decision bury each request in a particular machine’s queue.
On September 13, the project added turn-priority to its program-aware fairness policy on the main development branch. With a distinct fairness identifier for each conversation, this option favors deeper sessions to preserve their cached prefixes. Waiting adds to a newcomer’s score, but requests also have a time limit. Under the documented defaults, sustained competition from sufficiently deep sessions can keep a newcomer’s score below theirs until its waiting budget expires. New work can be rejected while established work continues.
This is a selectable policy, not an unavoidable property of AI or the plugin’s default strategy. The same plugin defaults to least-attained-service, which favors programs with less recent weighted token consumption, alongside waiting time. The alternatives embody different ideas about whose claim should grow: the conversation with reusable history, or the program that has received less service.
There is a practical case for protecting work already admitted. A service that keeps accepting new tasks while disrupting ongoing ones could leave everyone with an impressive collection of beginnings. Preserving a useful prefix also avoids spending scarce capacity on repeated computation. Favoring continuity can be an honest promise to users.
But the number of completed turns does not reveal how many remain. A deep conversation might be about to finish, or about to discover another branch of the problem. Cached context tells the scheduler something about the cost of the next request. It does not tell it the importance of the task or the distance to a useful result.
Giving admitted conversations priority may be worth a longer wait to start; sharing capacity more evenly may be worth some lost reuse. Neither promise follows automatically from cheaper prompt processing. The choice becomes especially consequential when an optimization can keep new work outside the queue’s effective reach. How should a service weigh the work it has already admitted against the person still waiting to start?