How vLLM and SGLang Serve Models Fast

0:00 One model, many waiting conversations

How vLLM and SGLang serve models fast. Picture this: you send a short question while someone else uploads a long document, and you both want an answer from the same model. A GPU is the computer hardware doing the model’s calculations. The problem is that making one answer fast isn’t enough if other conversations wait or the GPU sits idle. vLLM and SGLang are software systems for serving models to many requests, and the central idea is to coordinate the work and memory so the hardware can serve those conversations more effectively. The short question has a brief stretch of model work, and then its answer arrives a piece at a time. Those pieces stand for generated tokens, the small units of text a model produces. The important thing is that even a short answer takes repeated steps, while other requests may still be waiting. Now compare that with the much longer document. Its initial work takes longer, and its answer stream starts later than the short question’s. So requests sharing one model do not all need the same amount of work at the same time; that’s why serving has to coordinate them, not just run a fast model.

And there are more conversations waiting behind those two. The question is how to keep the model busy without making everyone wait longer than they need to. Okay, let’s compare the two activity lanes. The upper lane has unused gaps, while the lower lane packs the same illustrative work more tightly. That doesn’t claim a measured speedup; it shows why coordinating when requests get work can reduce idle time. “Fast serving coordinates work and memory.” That’s the idea we’ll carry forward, because the model’s calculations are only part of serving many conversations well. We still need to understand how a request gets from the outside into the model and back as an answer.

1:50 The route through the serving stack

First, let’s get clear on the serving basics, so we can follow what happens between a request arriving and its tokens being generated. Then we’ll look at parallelism, and how splitting work across GPUs can help serve more requests or run larger models. After that, batching and memory will show how requests share GPU work and make better use of limited space. Next, prefixes and chunks will show how reusing prompt work and breaking up long prompts can reduce waiting. Finally, advanced execution brings together speculative decoding, cache tiers, and separating prompt processing from generation. And with that route in view, let’s start with what happens inside a serving system.

2:34 Where the inference engine sits

A production request starts outside the model, with a client sending a question into the serving system. Here the client is the app or person asking for an answer, and this connector carries the request to the gateway. Okay, let’s shift to the gateway. It’s the front door around the model: it can check whether a request is allowed, choose where it should go, and decide whether to admit it or make it wait. That matters because an overloaded service needs a way to control incoming work. Now the request reaches the inference engine, the part that runs the language model and produces its answer. Its API server is the doorway requests use to ask for that work; an API is a set of agreed ways one program can ask another for something. vLLM and SGLang are two alternative engines that can offer familiar chat-completion interfaces, so an app can often use a familiar request format with either one.

That OpenAI-style badge means the interface follows a familiar pattern, not that every endpoint or option behaves identically. The next question is what happens to the request once it’s inside the engine. Inside the engine, the scheduler is the part that chooses which requests get GPU time and when. A job is simply a piece of work the engine needs to run; here, “Explain rain” is waiting for its turn. That gives us the handoff point between accepting a request and actually running the model. As the request moves into the running state, let’s look at what the GPU workers hold. A GPU is a processor built to do lots of number-crunching in parallel, which is useful for running a language model. The weights are the learned numbers that give the model its behavior, shared across requests using that model, while conversation state is temporary information kept for each request as it runs.

Now the generated tokens travel back along the path to the client. A token is a small piece of text the model produces at a time, so streaming means the client can start receiving the answer before the whole response is finished. Let’s add Metrics, the measurements used to track things like queue size, delay, memory use, and how much work the service completes. And notice the gateway’s admission limit: when demand is too high, limiting what enters helps stop a growing queue from making everyone wait longer. So the engine runs the model, but the gateway controls what gets admitted and metrics help track how the service is doing. The answer can stream back through the surrounding system, and production serving depends on all those roles working together.

5:08 The two very different kinds of model work

Okay, let’s start with the words in this little prompt. A token is one piece of text the model handles as a unit, though a real tokenizer—the tool that splits text into those units—can break a word into smaller pieces. So these whole words are just convenient toy tokens for us to follow. Real tokenizers can split a word into parts, so a token isn’t necessarily a whole word. And that’s the important difference—the real pieces the model handles can be smaller than the words we see here.

To understand the work, let’s bring in the transformer layers. A transformer is the model machinery that repeatedly updates what it knows about each token; these three layers stand for the repeated stages that do that. Okay, follow the arrows from “Explain.” That first position can use itself, but there isn’t an earlier prompt position for it to use. Now “the” can use “Explain” as well as its own position, so information can move forward through the prompt, but not backward from the future. And “rain” can draw on both earlier tokens and itself. Those arrows show causal attention, meaning each position can use only what has appeared by that point. That’s the rule to keep in mind: a position gets its own past and present, never a look at later text. It lets the model process a prompt without letting future words leak into earlier positions. With those dependencies in place, let’s see how the whole prompt moves through the layers together. The prompt positions advance together through the repeated layers, while each one keeps its own backward-looking attention pattern.

The supplied prompt is processed in a phase called prefill. The key idea is that all its positions can move through each layer together, with the causal restriction we just traced still in place. Once that prompt work is done, the model predicts its first new token. Here, that token is “falls.” Now the work changes. Decode is the repeated process of feeding the newest generated token back into the model so it can predict the next one. The reason we wait for this token to pass through the layers is that the next prediction depends on it being part of the model’s input. Now the next token, “softly,” appears only after “falls” has gone through the shown layers. So that’s the contrast: prefill handles the supplied prompt together, while ordinary decode grows one token at a time, with each new step depending on the token just produced.

Let’s shift from the layer-by-layer picture to what attention keeps around as tokens are processed. These two rows are keys and values, two kinds of vectors—lists of numbers the model uses inside attention. A key helps a new token decide which earlier positions are relevant; a value carries the information it can draw from them. Each layer has its own keys and values, so the paired rows here are a simplified picture of state kept separately throughout the layers. For “softly,” the model forms a query, which is like a search request for useful earlier information. It compares that query with the retained keys to decide how much attention to give each position. Then those attention amounts determine a weighted mixture of the values. In everyday terms, the query decides how much to take from each earlier token, and the values supply the information being combined. Keeping these keys and values is called the KV cache. A cache is a nearby stash of work already done, so the model can reuse it instead of rebuilding the old attention state at every generation step. Look at the rows as “softly” gets its own key-and-value pair. The older pairs stay put, so the new token adds to the retained attention state rather than making the model rebuild all the earlier pairs.

Now let’s use that token-and-cache picture to ask why prefill and decode can put different kinds of pressure on the hardware. A long prefill can give the model lots of token positions to work on in parallel. That often keeps the arithmetic units—the parts that do calculations—busy. With just one decode request, there’s much less parallel work at a time. The model may spend a lot of effort moving its weights—the learned numbers it uses—and the cached attention state through memory. But with several decode requests, there’s more work to handle together, as the wider band suggests. Decode’s resource demands can change when requests share the work. So the bottleneck really does depend on the workload. These bands show the breadth of available parallel work, not measured time or throughput, and we shouldn’t treat either tendency as a universal rule. Batch size, context length, model shape, and hardware can all shift which resource is under pressure. We’ve separated the kinds of work; the next question is how to fit a model across GPUs when one worker or one copy isn’t enough.

10:06 The memory budget behind concurrency

Okay, let’s start with the part of the GPU’s memory that stays put. These weights are the learned numbers the model uses to answer requests, and on one replica, all its requests use the same weights. That shared space is already spoken for, before we make room for anything else. And alongside those fixed weights, we need working space too. That’s temporary room the GPU uses while it’s doing the model’s calculations, so the memory budget has to leave space for both. Now we can separate the part that belongs to each conversation. Request A and Request B each get their own KV cache, the saved keys and values from earlier tokens that the model needs as it generates more text. The weights are shared, but this saved conversation state is request-specific. Let’s watch Request B’s token strip and cache region grow together. Each new retained token needs its own saved attention state, so longer live contexts consume more cache memory. The weights stay fixed, which means these growing conversations are what make the available room for other requests shrink.

So how do we estimate that growing cache? For every retained token in every layer, the model keeps two vectors, called keys and values. This first factor, two, counts those two saved vectors. Then we multiply by the number of layers, because each layer keeps its own keys and values for that token. And we count every retained token, since each one adds another set of saved keys and values through the layers. Next is the number of KV heads. A head is a group of key and value information used by the model’s attention calculation, so this counts how many such groups are stored. Each head has a particular width, meaning how many numbers make up its key or value vector. That’s why head width is another factor. Finally, each of those numbers takes up some number of bytes in memory. Multiply by bytes per number, and the whole product gives the memory needed for the cached keys and values. So the units come out as bytes of cached keys and values. We’re counting how many numbers are stored, then how much memory each number takes. One detail that changes the count is that some models let multiple query heads share KV heads. So the number of KV heads does not always match the number of query heads.

All right, let’s put numbers into that formula for this unsharded example. We have two stored vectors, thirty-two layers, eight thousand one hundred ninety-two retained tokens, eight KV heads, a head width of one hundred twenty-eight, and two bytes for each number. We’re counting one full, unsharded cache for that context. Multiplying those factors gives one billion seventy-three million seven hundred forty-one thousand eight hundred twenty-four bytes. That’s the cache estimate before we convert units. Now divide that byte count by one thousand twenty-four squared to express it in mebibytes, or MiB. The result is one thousand twenty-four MiB. And keep the scope of this example in mind. This estimate leaves out allocator overhead and cache sharing, so it’s an idealized unsharded cache calculation, not a complete GPU memory budget.

That cache estimate tells us how much memory a live context can use, but it doesn’t tell us how long a person waits for an answer. So let’s turn to the request timeline and separate the first response, the pauses between later tokens, and the amount the service finishes. The first wait starts when the request arrives and ends when its first output token appears. That whole interval is time to first token. You’ll often hear that shortened to T T F T. It’s still measuring the same wait, from arrival to the first token. Once the answer has started, we can look at the gap between one output token and the next. That tells us how quickly the answer continues to arrive. That gap is called inter-token latency. It measures the time between successive output tokens, not the wait before the first one. The third measure is throughput, how much work the service completes over a period of time. Here the counter is completed output tokens, counted across requests, and the fraction means we divide that total by the length of the observation interval. So production is not just about pushing throughput as high as possible. The goal is to complete plenty of work while keeping both the wait for the first token and the gaps during generation within their limits. And that connects the memory we’ve been counting to the service people actually experience.

14:41 Data parallelism replicates the service

The weights and memory limits we’ve just looked at make it worth asking how a service can handle more requests. This first replica holds the full model, so the model isn’t being split up here. Now the router has two requests, A and B, and two full copies to choose from. That’s data parallelism: each replica keeps the whole model, while different requests can go to different replicas. The copies are separate destinations, not pieces that need to work together on one request. As more requests arrive, some can go to Replica A and others to Replica B. The note gets at why we make these copies: with enough incoming work, they let the service handle more requests in total at once. That’s more capacity across the service, not automatically a faster answer for one person.

Okay, let’s look inside the replicas. Each one has its own local queue, meaning its own line of requests waiting for that copy, and its own local KV cache. The KV cache is a nearby store of attention state the model has already built for active conversations, so it can reuse that state as those conversations continue. These queues and cached state stay with their own replicas. And that’s the limit of this kind of scaling. Adding another full copy doesn’t make the model in either copy smaller, and it doesn’t by itself make a single answer arrive faster. Each replica still needs room for its full weights and the conversation state it’s handling. In production, an external router can sit in front of replicas running vLLM or SGLang. Here, request C is assigned along the connection to Replica A. The router chooses a destination; the replicas remain separate full copies.

Both vLLM and SGLang offer ways to run multiple independent replicas for data parallelism. The label here means either engine could serve this replica role; they aren’t two extra services in the request path. The pending row changes as requests are assigned, but the key idea stays the same: each request is served by one independent replica. The exact launcher and routing features depend on the release and how the service is deployed. So, let’s bring in the measurements that help the router choose. Metrics is the readout for each replica’s queue and latency, and load information connects it to the router and both replicas. That gives the router more to go on than simply counting requests.

Now compare the work shown in these two replicas. A long prompt with many tokens can require much more prompt processing than a short question, so one request in the queue doesn’t always mean the same amount of work as one request in the other. There’s also a new routing criterion here, cache affinity. It means preferring a replica that already has reusable state for a request’s prefix, when that saved work is worth it. The label raises that possibility; we’ll need to weigh it against the replica’s load.

17:36 Tensor parallelism splits one model operation

Okay, we’re shifting from sending different requests to different model copies to splitting one model’s calculation across GPUs. A matrix is a rectangular set of numbers, and multiplying it by an input is one of the calculations a model repeats. The input here is one and two, and we’ll divide the matrix’s output columns so the GPUs can work on different parts of that same calculation. Now the matrix is split into two column groups. GPU A owns the first two output columns, while GPU B owns the last two, and the same input stays shared. That means each GPU can work out its own piece of the output rather than producing a separate prediction.

Let’s follow the first column owned by GPU A. We multiply the matching input components, one times one and two times zero, then add those products to get one. For GPU A’s second column, the same input gives one times zero plus two times one, which adds up to two. So GPU A has its two output components, one and two. They belong together as the first piece of the result. Now we’re on GPU B’s first column. Using the same input, one times two plus two times one gives four. For GPU B’s other column, one times one plus two times two gives five. GPU B’s piece is four and five. Together, the two GPUs’ pieces make the four-component output, with each piece still associated with the columns that produced it.

The four dot-product calculations have done their job, so we can set that working aside. ReLU, pronounced “ray-loo,” is a simple operation that keeps each positive number and turns any negative number into zero. Let’s apply it to each GPU’s output piece. GPU A’s components are already positive, so ReLU leaves one and two unchanged. That’s why this step can be done right where those components are, without combining the GPUs’ pieces. GPU B’s four and five are positive too, so its activation piece stays four and five. Now the two activated pieces are named h A and h B, while the completed first multiplication stays summarized as one four-component result. The next matrix is divided by rows, matching each row shard to the activation piece on that GPU. That pairing lets each GPU compute part of the next output locally. Here the split of the next matrix follows its input rows. GPU A gets the first two rows, and GPU B gets the last two, so each activation piece lines up with the rows it needs for its local multiplication.

Let’s start with GPU A’s first output component. Its activation, one and two, multiplies the first column of its row shard, one and zero, giving one. For GPU A’s second component, the column is zero and one, so one times zero plus two times one gives two. GPU A has now collected its two contributions into a partial vector, one and two. It’s only one part of the final answer, because the other GPU’s row shard also contributes to each output component. Now let’s follow GPU B’s first contribution. Its activation is four and five, and the first column of its row shard is one and two. Four times one plus five times two gives fourteen. For GPU B’s second contribution, the column is two and zero. Four times two plus five times zero gives eight. GPU B’s displayed partial vector is fourteen and four. We’ve now got two partial vectors, and each position in the final output still needs the contributions from both GPUs. Here’s the element-by-element addition: one plus fourteen gives fifteen, and two plus four gives six. All-reduce is the name for the communication step that combines those partial results and makes the combined result available to the participating GPUs. Now the same combined output, fifteen and six, is shown at both GPU A and GPU B. Each GPU has its local contribution, and the collective step gives both the completed result they need.

Let’s trade the detailed arithmetic for a pattern we can follow across both ownership lanes. Each GPU does local work, and then the lanes meet at a combine stage before local work continues. In a real model, similar patterns show up when large matrices or attention heads are split across GPUs. The takeaway on screen is that more shards also create communication work. Splitting the calculation can share the local work, but the GPUs still need to combine results along the way, so the communication cost matters too.

22:24 Combining data and tensor parallelism

We’ve split one model’s work across GPUs before; now let’s see how that cooperation sits inside a serving setup. The two GPUs in Replica A are joined by a collective link, which lets them coordinate while working on the same request. Okay, as request A comes in, notice that it appears in both A1 and A2. These are not two independent copies answering the same question; they’re the two workers of one replica sharing the work. And now request B goes to a separate pair, Replica B. Its workers cooperate with each other, but this path is independent of Replica A’s, so the router can send different requests to different replicas.

Let’s shift from which request goes where to what each pair is doing together. Each GPU now holds a weight shard, meaning a portion of the learned model numbers rather than a whole copy. The links are labeled NCCL, a library GPUs commonly use to exchange and combine data during collective operations. vLLM and SGLang both use tensor-parallel groups like these, where the workers coordinate on a model’s computation. So the point of splitting the weights is that, for a compatible model, the full set of weights can fit across the group even when one GPU cannot hold it alone. That makes the model available to run; it doesn’t tell us any particular memory total. There’s another piece to check besides the weights. KV cache is the saved attention state for the tokens a request has already processed, kept so the model can reuse it as it continues. Some of that state can be divided across workers, but whether it divides neatly depends on the model’s layout.

Here, the weights are still sharded, while the cache entries are labeled with the same head, h zero, in both A1 and A2. A head is one group of the key and value state used by attention. This repeated label shows why a small number of KV heads may need to be replicated in some tensor-parallel layouts, rather than each worker getting a distinct head. And that repeated head changes what we can infer about memory. Splitting the GPU work across more workers doesn’t guarantee that every part of memory, especially the KV cache, shrinks by the same factor. So how do we choose between more independent replicas and a larger cooperating group? These highlighted request paths represent the first option. If one model already fits on a GPU, additional replicas can serve separate requests and may deliver more total throughput than putting those GPUs into one tensor-parallel group.

But if the model does not fit on one GPU, the collective links matter for a different reason: the weights have to be split across workers for the model to run. That can make tensor parallelism essential, even though it brings coordination within each group. The right balance depends on the actual model, the connections between GPUs, and the latency you’re trying to meet. So the useful choice is the one that performs well under workload measurements, not simply the one with more replicas or more workers.

25:30 Continuous batching fills the next available slot

Okay, let's set up the comparison. A, B, and C are already ready to generate, with three, one, and two output tokens left. We’ll follow what happens when each active request gets one token per iteration, and see what changes when a request finishes. All the prompts are already processed, so this trace is only about generating answers. Each occupied slot produces one token in an iteration, then we update how many tokens that request still has left.

Let’s take the fixed batch one iteration at a time. A goes from three tokens left to two, B goes from one to zero, and C goes from two to one. B has finished, but in a fixed batch its slot stays unused while the original group continues. And now D appears in the waiting area with two output tokens left. The timing matters here: D becomes available after this first iteration, but this fixed batch still keeps its original requests together. A has one token left after this iteration, while C reaches zero and finishes. So now two of the three slots are unused, even though A is still generating and D is waiting. A’s remaining count reaches zero here, so the original batch is finally done. Its other two slots have already been sitting unused, but D still hasn’t joined; the fixed batch waits until the whole original group finishes. Only now does D get admitted, in iteration four. That’s the cost of holding the requests together: D waits even though a slot became free much earlier. D uses one of its two output steps here, leaving one token to generate. The other two slots remain unused. D finishes in iteration five. So even in this tiny example, the fixed batch stretches the trace out to five iterations, with several slots idle along the way.

Okay, let’s compare that with continuous batching underneath. We keep the same requests and the same remaining output lengths, but now the group can change at an iteration boundary instead of staying fixed. The first iteration has the same outcome: B finishes, while A and C still have work left. So there’s an available slot before the other requests are done. D is available now, and the vacated slot can take it for the next iteration. That’s the key change: the scheduler can admit a waiting request as soon as a slot opens, rather than waiting for the whole batch to finish. In iteration two, A has one token left, D has one token left, and C reaches zero and completes. The active group has changed, but generation keeps moving forward. Now A and D each use their final output step, leaving the third slot unused. Both finish in this third iteration. Here’s the comparison: with the same stopping lengths, D finishes in iteration three under continuous batching, rather than iteration five in the fixed trace. The other completions line up too: B finishes in iteration one, C in two, and A in three. The slot freed by B let D start earlier, so fewer slots sat idle while work was waiting. But what can a real scheduler admit at any given boundary? That depends on its resource limits, and a new prompt may not yet be ready for decoding.

The next admission isn’t automatic; it has to fit three limits. KV capacity is the GPU memory needed to hold the attention state for active requests, so longer or more numerous conversations can use more of it. The token budget limits how much token work the scheduler takes on in one iteration. And the sequence limit caps how many requests can be active at once. There’s another distinction to keep in view. A new request marked “Needs prefill” still has prompt processing to do; it isn’t ready to join this decode-only trace yet. Prefill processes the prompt so the model can start generating from it. So vLLM and SGLang schedule work at iteration boundaries, but admission still depends on available capacity, token work, and active-sequence limits—and on whether a request has finished prefill. Continuous batching keeps the door open for new work; the scheduler still has to decide when that work is ready and fits.

29:45 PagedAttention separates logical order from physical memory

Okay, the question here is how to keep each conversation’s cached attention state without reserving a huge, fixed stretch of memory for an answer whose length we don’t know yet. This pool stays fixed, and each block has four token slots, so we can see what happens when requests take only the space they need. As these outlines mark out large contiguous reservations, notice how much capacity they set aside before we know whether either request will use it. That’s the waste we’re trying to avoid. Let’s replace those oversized reservations with the actual retained state. A has six tokens, so its logical order spans blocks three and zero, while B’s three tokens go in block two. The physical blocks don’t need to sit next to each other for each request’s tokens to stay in order. And each slot here stands for a key and value pair from the attention state, not a piece of raw text. That’s what this pool is storing as the conversation grows.

Okay, now let’s turn to the block tables, the little maps that connect each request’s logical blocks to physical blocks in the pool. A maps logical block zero to physical block three, then logical block one to physical block zero; B maps its first logical block to physical block two. The token order stays logical, while these entries tell the system where the state actually lives. We’ll use A’s token at position four as a lookup example. Positions start at zero here, so that label identifies the fifth token in A’s sequence, not the fourth one. With four slots per block, divide position four by four to get logical block index one, and take the remainder to get offset zero. One part tells us which logical block to look up; the other tells us which slot inside that block. So the table’s entry for logical block one points to physical block zero, and offset zero picks the first slot there. That’s how the attention kernel finds A’s selected key and value state without moving the pool or needing one long, contiguous stretch.

Now A gets two more token states. Watch how they use the free slots at the end of its existing final block, so the table doesn’t need to grow yet. Those two additions fill A’s current final block. A still needs to keep growing, so its table now gains an entry mapping logical block two to physical block one. A gets one more state, and it goes into that newly allocated physical block. So the request now has nine tokens, with its logical blocks mapped to physical blocks three, zero, and one. B is finished, and its unshared block can go back into the free pool. The useful thing is that the pool itself stays put; it’s the allocation and the table entries that change. C can now use physical block two, the same block B released. C’s two states occupy only part of it, and A’s blocks remain in their original locations.

Block size trades flexibility against overhead. Smaller blocks can leave less unused space at a request’s end, but they also mean more block boundaries and table entries to manage. Larger blocks simplify that bookkeeping, while a partially filled final block can leave more capacity unused.

Different kernels can use the same memory-management idea. A kernel is the GPU routine that carries out work such as attention; the key idea here is that it can follow a mapping from logical blocks to physical storage. Here, vLLM’s PagedAttention and SGLang’s KV pools are shown beside one unchanged pool and mapping. The labels name different engine approaches, not two competing pools or a claim that both use the identical attention kernel.

Let’s turn to sharing. D now points to the same full prefix blocks as A, and this example also has both requests referencing the same partly filled tail. That shared state is useful because the common history doesn’t need a separate copy for each request. The full prefix blocks are marked as immutable, meaning they stay unchanged while both requests use them. The partial tail is a separate illustrative case; because it may need to change as a request continues, we need to keep its shared status distinct from those fixed full blocks. Before D writes to that shared tail, its contents are copied into physical block four. The copy gives D a place to continue without changing the original tail A still references. Now D’s table points logical block two to physical block four instead of block one. The common full prefix entries stay the same, while D’s tail has its own writable block. D’s next state, D nine, is added to the copied tail. A’s tail and the shared full prefix remain unchanged, so only D’s continuation grows here. So the full prefix remains shared, but the writable tails are separate: A’s is in block one, and D’s is in block four. That lets both requests reuse their common history while continuing independently.

34:48 How the engines keep the GPU fed

Okay, this block-table work sits inside a bigger loop. The waiting requests are what the scheduler has to choose from, and the question is how it decides which work can safely run. Now req A has been admitted only after the KV manager reports “Allocated.” The KV manager keeps track of the memory needed for each request’s attention state, so the scheduler checks capacity before sending a batch to the GPU executor. That way, work is submitted only when there’s room for it. And req B is marked “Waiting for capacity,” so it stays queued rather than taking memory the GPU doesn’t have available. That’s the control loop doing its job: requests can wait until capacity opens up instead of overrunning memory.

For vLLM V1, the scheduler tracks a token budget—the amount of token work it can schedule—and the work that will go into the next batch. It coordinates that choice with block allocation, which reserves KV memory in manageable pieces. So scheduling work and making room for its cached state have to agree. As that work reaches the GPU executor, supported model kernels—the routines that carry out the model’s calculations—and an attention backend—the code that computes attention—do the actual GPU work. The token appearing in Output is the result coming back from that execution. Here’s the useful distinction: paged allocation decides how the KV memory is organized, while fast attention is the calculation that uses it. They work together, but they’re different jobs, so improving one doesn’t make the other disappear.

The component frame stays the same, but the engine label changes to SGLang, and the KV manager now includes a radix cache. A radix cache organizes reusable prompt prefixes so the system can find prior attention state instead of rebuilding matching prefix work. The scheduler still chooses work, and the manager still handles its memory. Now the GPU executor is handling the current batch while the scheduler prepares the next one. That overlap means CPU-side scheduling can get ahead while the GPU is busy, where the execution path supports it, rather than waiting for one batch to finish before beginning all preparation for another. One more piece of the loop is how the GPU receives its work. Both systems can use specialized GPU kernels, and on supported paths they can also use captured execution graphs to reduce the overhead of repeatedly launching the same work.

A captured execution graph is a saved recording of GPU work submission. When the work matches, the system can reuse that recording instead of rebuilding the launch sequence each time. But this request is marked “Unsupported shape,” so the executor has switched to a fallback path. Shape means the arrangement and sizes of the work the model is handling; if that arrangement doesn’t fit the captured path, the graph isn’t used. So these optimizations depend on the execution path available for the particular request.

37:48 Prefix caching reuses a past computation

We want to see why a later request sometimes doesn’t need to redo all the model’s work. Here, R1 is the first request, and each word in its strip stands for one toy token, a piece of text the model processes as a unit. As the first token, “You,” is processed, the model retains its key and value state. Those are the pieces of attention information later tokens use to look back at earlier text. Now “are” gets its own retained state too, so the model can build the request’s attention history one token at a time. The same happens for “helpful.” This state belongs to the processed prefix, not to any words that might come afterward. And “Explain” adds another piece of retained state to R1’s growing history. Finally, “stars” is processed as part of R1, with its own key and value state. Okay, let’s compare R2 with R1 from the very first token. Their first four tokens match exactly, and only the last one differs, with “stars” in R1 and “rain” in R2. That exact shared beginning is what might let us reuse work. Here’s the prefill comparison. Prefill means processing the tokens already supplied in a request before generating an answer. R1 has run through all five positions; for R2, the first four are marked for reuse, while “rain” is still waiting to run. The question is whether the retained state for that shared beginning really stays valid when the ending changes.

The key is causal attention, which means each token can use the earlier tokens but not future ones. So the state for “You are helpful Explain” doesn’t depend on whether the next token is “stars” or “rain,” and that makes the exact shared prefix reusable. Now only R2’s uncached “rain” position runs, and it gets a private KV segment. The four shared prefix segments stay fixed, so the reused state remains separate from R2’s new continuation. That saves work on the supplied prompt, but it doesn’t generate the answer for us. Future output tokens still have to be produced one by one. So the useful result is that a prefix hit saves prefill work: R2 reuses the four matching positions and processes its own remaining prompt token.

Now let’s turn from the work timeline to how the engine can organize these reusable prefixes. A radix tree is a branching index for token sequences; to keep it compact, one edge can hold a whole run of tokens instead of giving every token its own separate edge. Here, “You are helpful Explain” is one shared run, with its shared KV reference. At the end of that shared run, the paths separate. R1 continues along “stars,” and R2 along “rain,” with a distinct cached reference for each branch.

Let’s see what happens when a request shares less of the beginning. R3 starts “You are concise,” so we can follow its tokens against the existing path and see where the match stops. The match ends at “You are.” The tree splits the old shared run there, leaving “helpful Explain” on the original path, so its “stars” and “rain” branches remain intact. That’s how one compact tree can retain prefixes of different lengths rather than treating each request as an unrelated whole string. R3 then adds “concise” as a sibling branch from that shared beginning. Its cached continuation is separate from the preserved “helpful Explain” subtree.

Now let’s test a different kind of match. This request begins with “We,” so its first token differs from the tree’s “You” path. Even though the later words look familiar, the lookup stops right at the start. That early difference changes the attention history for the later tokens. Matching a few later words by themselves doesn’t make them a reusable prefix, because their state was formed in a different preceding context. There’s another requirement too. The cached state has to come from compatible model state and cache settings; otherwise it may not represent the computation this request needs. Let’s return to the original matching R2 request. Its final token, “rain,” is marked as a possible boundary position, even though the request’s prefix state is available. Here’s the subtle point. A cache may retain the keys and values but not the next-token prediction produced from the final position. Depending on what the engine cached, it may need to execute that boundary token to obtain the prediction for what comes next.

42:21 Two ways to index reusable KV state

Okay, the tree showed us how matching works. Now we’re comparing how vLLM finds reusable state: it looks up whole blocks of a request’s KV cache, the saved attention information the model would otherwise need to rebuild. That full block is the reusable piece sitting in the KV pool. Keeping it there means a later request can reuse the saved attention state instead of doing that prompt work again. Let’s follow what makes the block key specific. It combines the block’s tokens with the identity of the prefix before them, so identical tokens after different histories do not get treated as the same cached computation. The lookup goes to the index, which then points to the stored block; we’re describing the key’s ingredients, not calculating a cryptographic hash.

Now let’s switch the index to SGLang. Its prefix-reuse method is called RadixAttention, which uses a tree-like index to find matching runs of tokens. The token path is the route through that index, and the KV references point to the saved attention state in the pool. So the index changes, but the pool still holds the actual reusable state. And the index reports the matched prefix to the scheduler, which uses that information to plan the request. The tree helps find reusable state; it does not replace the attention calculations or the memory holding that state.

There’s another practical question once the pool fills up. Both engines need a way to decide which saved state can go when memory is needed. Let’s keep the state used by a running request separate from spare cached state. If a request still refers to a block, that block has to remain valid while the request is running. Now Block B is marked evictable, while Block A stays in use. Evictable means this saved block is no longer needed by a running request, so it can be removed if capacity is needed, even though it might have helped a future request.

Block B is gone, and its reference is gone too; Block A and its reference remain. That frees room for new state without breaking the request that still depends on Block A. Freeing space isn’t the only concern. A cache lookup also has to avoid handing one request state produced for an incompatible execution, because reusing the wrong state would make the saved computation invalid. Let’s unpack these identity labels. A model is the set of learned weights doing the language task, and an adapter is an extra set of learned weights that changes the model’s behavior. A namespace is a way to keep one group’s cache separate from another’s; these are request identity fields, not a universal recipe for a cache key.

Here the namespace differs, so the lookup is denied rather than returning the existing reference. The block stays in the pool; separating namespaces lets a deployment enforce which requests may reuse one another’s cached state. With several model replicas, a saved prefix may be available on one replica and absent from another. So the router has to choose not just where the request can run, but whether reaching a local cache hit is worth steering toward. Let’s weigh the two things shown here, load and cache affinity. Load means how busy a destination is; cache affinity means a preference for a replica that already holds useful prefix state. SGLang’s routing ecosystem includes cache-aware routing, and vLLM deployments can use cache-aware routers or shared cache infrastructure.

And that readout captures the tradeoff: keep the cache hit only when the work it saves is worth any extra queueing or transfer. A less-busy destination can be the better choice, even if another replica already has the prefix.

46:05 Chunked prefill keeps long prompts from monopolizing execution

Okay, here’s the problem we’re trying to solve. A long prompt can take over the work for a while, even as two other requests are waiting for their next token. A token is one piece the model processes at a time, and chunked prefill means breaking the prompt’s initial processing into smaller pieces so other requests have more chances to run. So instead of one uninterrupted stretch, the prompt is divided into chunks, while D1 and D2 are still waiting. That creates opportunities to fit prompt work around ongoing generation.

Let’s make that scheduling decision concrete. We’ll use an illustrative budget of eight token positions per iteration, with fifteen uncached positions in P. An iteration is one round of scheduling work, and uncached positions are prompt pieces that still need processing. The green strip is where the prompt’s key and value state, used to preserve its attention history, is kept as the chunks are processed. First, D1 and D2 each get one position, so those two decode requests keep receiving a turn while P is being processed. With two of the eight positions reserved, six remain for P. The scheduler then compares that allowance with the fifteen positions still waiting, and uses whichever amount is smaller. That gives the first chunk positions one through six. They use all six places left after the decodes, so the prompt moves forward without taking the whole iteration away from them. Subtract those six processed positions from the original fifteen, and nine prompt positions remain to carry forward. Those first six positions now have their key and value state in the prompt cache, ready to support attention as later parts of the same prompt are processed.

For the second iteration, D1 and D2 each reserve a position again, leaving the same six-position allowance for P. Nine prompt positions remain, but only six fit in this iteration’s allowance, so the next chunk uses six. That’s positions seven through twelve. They follow the first six in order, keeping this as one continuing prompt rather than treating each chunk as a separate prompt. After processing those six, three of the original fifteen positions remain. The key and value state for positions seven through twelve is added to the same prompt cache, after the state from the first chunk. Now only three prompt positions remain, so even though six positions are available after reserving the decodes, P uses just three. Subtract the final three from the three remaining, and no uncached prompt positions are left. The final chunk’s key and value state joins the same cache, completing the processed prompt. And those backward attention links show why the chunks still belong to one prompt. Later positions can attend to earlier cached chunks, so dividing the work across iterations preserves the prompt’s causal history.

Let’s compare two ways to divide that same fifteen-position prompt. The finer partition is six, six, and three, while the coarser partition keeps all fifteen together. Smaller chunks give the scheduler more opportunities to serve decodes; larger chunks can make prompt processing more efficient, but can leave those decodes waiting longer. So the chunk size trades responsiveness against execution efficiency. The budget counts token positions, though, not equal amounts of time, and a decode position in a long context can be expensive to process.

Let’s connect the schedule to prefix reuse. A prefix is the beginning of a prompt, and when that beginning has already been processed and cached, the request can reuse it. The chunks shown here are the uncached remainder. Both vLLM and SGLang are labeled on this shared scheduling mechanism. The labels identify the engines using it, not a performance comparison. The order matters: reuse the available prefix first, then schedule only the remaining prompt work in chunks. That way, prefix caching avoids repeated processing, while chunking helps the remaining work share execution with ongoing decodes. The exact defaults and supported combinations vary by release, so the specific behavior depends on which version is being used.

50:24 Hierarchical caching buys capacity with movement

The chunked prompt still needs its reusable prefix, the part it has already processed, to continue without doing that work again. The prefix index is the lookup guide here; it records where that cached state can be found. Okay, let's follow this old chat as it moves out of the GPU cache. GPU memory is the fast, limited working space the model uses while serving requests, so keeping every conversation there would leave less room for active work. Now the old chat is in the host cache, which means the computer's main memory, while the index records its location as Host. Storage cache is a colder tier for state we may want to keep but use less often; it can hold more, though getting state back takes movement through the tiers. The useful tradeoff is that hot state stays close to the GPU, while older state doesn't have to occupy its scarce space.

A lookup finds the old chat in host memory, but that doesn't mean the GPU can use it yet. The scheduler marks the request Restoring, so the system knows it still has to bring the cached state back. The GPU cache now has a reserved spot for the returning prefix, alongside the active request that's already there. That space has to be set aside before the host copy can be restored, and the request needs to wait until the transfer is complete. Now the old chat is resident in the GPU cache, the index points to GPU, and the request is Ready. The active request stayed in the GPU cache too, so restoring one prefix did not mean removing that unrelated work.

All right, let's turn to how SGLang organizes these locations. HiCache extends SGLang's radix cache, its index for finding shared token prefixes, so it can track cached KV state across tiers. KV means the saved key-and-value attention state that lets the model reuse earlier prompt work. Here, the index is labeled Radix locations, and the backup path is shown from GPU toward host. The older chat is now shown in the host cache, arriving through the prefetch path from storage. Prefetch means bringing colder state closer to where it may be needed, rather than waiting until a request is already stalled for it. That gives the system a chance to prepare state ahead of use. The index now records the old chat at Host, and the storage entry is empty. That location tracking has to follow the actual movement; which paths are available depends on the configured backend and the SGLang release.

Now let's compare the vLLM view. Its Prefix cache is the built-in, in-engine place to reuse matching prompt state, so a request can avoid processing a prefix that is already available there. For movement beyond that local cache, the KV connector is the interface that lets supported integrations move cached state between tiers. The host tier is shown as part of that optional path, not as something the prefix cache automatically supplies by itself. LMCache is a separate cache system that can be configured to provide additional tiers, as this optional example shows. Simply installing vLLM does not mean LMCache, storage, or every backend is already set up; those pieces have to be configured for the deployment.

So when a request needs a prefix, the scheduler has two paths into the same GPU destination: restore the saved state from host, or recompute it. The useful question is which path gets the request ready sooner. For this example, the host entry is a large prefix, and the GPU is marked as available to receive it. Transferring a large prefix can save substantial repeated work, while rebuilding a small one may take less time than fetching it. The restore path also depends on how quickly data can move, whether other transfers are competing for that capacity, how the cached state is represented, and whether there's room for it in GPU memory. Those conditions affect the cost of restoring, not just whether a cache entry exists.

And that's the decision in one sentence. A cache hit helps only when retrieving its state is worthwhile compared with recomputing it, within the time the request is allowed to wait. The tiers add capacity, but they don't make every hit automatically useful.

54:39 Speculative decoding verifies a guess

Okay, we’re looking at a different way to reduce the cost of generation. Instead of asking the expensive target model for one token at a time, a cheaper draft process can suggest a short run, and the target checks those suggestions before they’re committed. Let’s start with the accepted prefix, the text we already know is verified, and watch the draft add Red as a provisional guess. Provisional means it’s only a suggestion so far, not part of the committed output. Then Blue is proposed after Red, still provisionally. The draft is building a sequence one token at a time, but neither suggestion is committed just because the draft made it. Now both guesses are there, and the target has a distribution at each position to check them. The useful idea is that it can score the whole proposed sequence in one verification pass, rather than calling the target separately for each guess.

The important detail is the one-position shift. The distribution after the prefix checks Red, and the distribution after Red checks Blue. That’s how causal attention works here: each token is judged from the history before it, not from a prediction that uses the token itself. So when the sentence says verification uses the preceding position, it means exactly that alignment. Red is checked from the prefix; Blue is checked from the prefix with Red added.

Let’s turn from where the target gets its predictions to how a guess is accepted. The vocabulary here is Red, Blue, and Green; for the first check, the draft distribution, written q one, assigns them six-tenths, three-tenths, and one-tenth. The target distribution, p one, assigns them five-tenths, three-tenths, and two-tenths. The acceptance rule uses the target’s probability for the proposed token divided by the draft’s probability for that same token, with the result capped at one. That way, a proposal can be accepted often when the target considers it more likely, but the chance never goes above certainty. For this check, the proposed token is Red, and the prescribed uniform draw is zero point four. A uniform draw is a random number between zero and one that we compare with the acceptance probability; we’ll use it once that probability is worked out. For Red, divide the target’s five-tenths by the draft’s six-tenths. That gives about zero point eight three three three for the ratio. That ratio is already below one, so capping it at one leaves the acceptance probability at about zero point eight three three three. The draw, zero point four, is less than the acceptance probability, about zero point eight three three three, so Red is accepted. And now it moves into the committed output; Blue is still only a provisional guess.

With Red accepted, we check Blue using the distributions for that history. For Red, Blue, and Green, q two is three-tenths, four-tenths, and three-tenths; p two is five-tenths, two-tenths, and three-tenths. So this second check is conditional on having accepted Red first. Now the proposed token is Blue, and its uniform draw is zero point eight. We compare that draw with Blue’s acceptance probability. For Blue, the target probability is two-tenths and the draft probability is four-tenths. Their ratio is zero point five, and capping at one leaves the acceptance probability at zero point five. Zero point eight is greater than zero point five, so Blue is rejected. Red stays committed, while Blue’s provisional state has to be discarded; we stop accepting this draft here.

Okay, Blue was rejected, so we need the correction rule rather than simply keeping the guess. The correction starts from the target probabilities minus the draft probabilities, separately for Red, Blue, and Green. Subtract component by component. For Red, five-tenths minus three-tenths is two-tenths; for Blue, two-tenths minus four-tenths is negative two-tenths; and for Green, three-tenths minus three-tenths is zero. A correction distribution cannot have negative probability, so we replace that negative Blue component with zero. The positive residual is two-tenths for Red, zero for Blue, and zero for Green. Now add those positive residuals to get a normalizing sum of two-tenths, then divide each component by that sum. The resulting correction distribution has components one, zero, and zero, in Red, Blue, Green order. The uniform draw of zero point five selects Red from this correction distribution. So Red is committed as the correction, while rejected Blue and its incompatible provisional state are removed. This residual step is what preserves the target model’s sampling distribution; an ordinary replacement sampled from p would not, in general, do the same job.

Let’s leave that numerical rejection trace and look at the separate case where every proposal is accepted. Here A and B are just abstract token labels, so we can focus on what happens to the sequence without adding new probabilities. When all the proposals are accepted, the target can also supply a bonus next-token sample after them. That gives the sequence one more target-generated token in the same verification pass. There’s also a shorter alternative called greedy matching. It uses a matching rule for greedy output, rather than the sampling arithmetic we just worked through. And exactness depends on the verifier being correct. The point is not to emit unchecked draft guesses; the target’s verification and correction rules are what keep the output faithful to the intended target behavior.

So what does all this buy us, and what does it cost? A speculative cycle has three parts: draft proposals, verify them with the target, and commit only what has been verified or corrected. Compare these outcomes. With high acceptance, several verified tokens can share one target call. With low acceptance, rejected proposals still cost draft and verification work, so less of that work turns into committed output. There are costs even before a rejection: the draft takes time and memory, and the target processes more positions during verification. The benefit depends on whether those costs are outweighed by committing multiple tokens per target call. And if a large batch is already keeping the target busy, speculation may add little benefit, because the target call is no longer the same kind of bottleneck. So more proposed tokens do not guarantee more speed. What matters is whether enough guesses are accepted to save target calls without the draft and verification costs eating up that gain.

61:15 Speculation inside vLLM and SGLang

Okay, the target verifier sits between proposals and output. It checks the proposed tokens, and only tokens it accepts go on to become committed output. Here the proposer is a separate, smaller model. It makes draft tokens for the target verifier to check, so the larger target model doesn’t have to come up with every candidate itself. And this is a different approach. A trained drafting head uses features from the target model to help propose tokens, rather than relying on a separate small model. That lets us distinguish feature-based drafting from the draft-model option.

For vLLM, the proposer can take several forms, but which ones work together depends on the supported configuration. A draft model is one option; prompt n-grams and EAGLE are others, and some model-specific configurations use multi-token prediction, or MTP, to propose multiple tokens. These are supported choices, not a promise that every combination works. Now the proposer is using a prompt n-gram. An n-gram is a short run of tokens, and here the proposer looks for a run already found in the prompt to suggest what might come next. It’s a lightweight way to propose continuations without introducing a separate draft model. EAGLE is the feature-based option here. Its trained drafting method uses features from the target model, rather than simply running an arbitrary small model as the drafter.

SGLang also has supported speculative paths, including EAGLE and EAGLE-3. The proposer choices depend on the execution configuration; that short label doesn’t mean every draft method is available in every setup. Here the proposer holds a candidate tree, so it can offer several possible continuations branching from the same prefix rather than just one chain. The target verifier’s matching tree-verification layout lets it check those branches while respecting which tokens depend on earlier ones. That’s the integration point: the verifier has to match the shape of the proposals. As the verifier decides how many proposed tokens are accepted, the KV manager has to keep track of which state is committed and which is still provisional. KV state is the saved attention information the model needs to continue a conversation; provisional state belongs to draft tokens that have not yet been accepted.

Now the provisional row is marked rejected. That state must not carry into the next request as though those tokens had been accepted; only the committed state remains valid to continue from. And the note on the verifier says support is specific to the execution configuration. So before deploying, check the exact release, model, quantization mode, parallel layout, and feature compatibility. Then measure how often proposals are accepted, how much drafting costs, and the end-to-end latency under real concurrency. Which proposer is useful depends on that actual combination, not just its name.

64:13 Prefill-decode disaggregation separates resource pools

A request comes in from the client, and the router is the traffic director deciding where that work should go. We’re changing the setup here from speculation, which changes how tokens are proposed, to disaggregation, which separates prompt processing from generation. Okay, now let’s follow the prompt job into the prefill pool. Prefill means processing the supplied prompt so the model can build the attention state it needs; keeping that work in its own pool separates it from the repeated steps of generating an answer. And here the router reserves room in the decode pool for this request’s generation. The model weights are loaded in both pools, but their jobs differ: prefill handles the prompt, while decode will continue the answer once the needed state is ready.

Let’s shift to what prefill has produced. The KV cache is the saved key and value information for the prompt’s tokens; keeping it means decode can reuse the model’s attention work instead of processing that prompt all over again. Those entries stay inside the prefill pool as the handoff begins. The decode pool is reserved, but it still needs the prompt’s KV state. Text and a request identifier tell it what was asked, not the attention history the model already built, so without that saved state it would need to recompute the prompt. Now the KV transfer is underway, and the destination changes from reserved to receiving. The large payload is the cached key and value tensors; the smaller metadata tells the receiving side how to interpret that payload. Both matter for continuing from the same prompt state.

The decode pool is marked ready once the handoff has made the required state available. There’s also a boundary detail to preserve: the receiving side needs to know where the prompt ends and generation continues. One detail can vary between protocols: some produce the first token on the prefill side before handing off, while others draw that boundary differently. That’s why we shouldn’t assume one universal first-token rule. Now the decode pool is generating, and the tokens travel through the router to the client as a stream. The next token continues the same model history, so the boundary has to avoid both skipping a token and generating one twice.

With the two pools separated, each can be scaled and configured for its own kind of work. Prefill can be tuned for processing prompts, while decode can be tuned for continuing active answers, and the pools can use different batching policies or hardware choices. The decode pool now has its own queue. That separation can keep a long prompt from interrupting the same workers that are producing tokens for active answers, though the queue still needs to be managed. Here’s the tradeoff at the transfer layer. Separate pools need their own queues and enough model capacity on both sides, and the KV payload has to cross a handoff before decode can use it. So isolation can help separate the workloads, but it adds work on the critical path.

For vLLM, the KV connector is the interface that lets the serving engine hand cached state between the prefill and decode sides. This transfer layer shows NIXL and LMCache as alternative integrations, not two services that must be chained together. Now the label changes to SGLang transfer. Mooncake and NIXL are alternative backends here too, ways to carry the KV state across the handoff rather than mandatory stages both requests must pass through. And that’s the important qualification: these names are configurable transfer infrastructure. Which backend and topology are available depends on the version and deployment, so the labels aren’t a promise that every setup supports every combination.

Before bytes move, the two sides need layout agreement. They have to be using compatible model state and agree on how the KV entries are arranged, or the receiving side won’t know what those entries mean. Ownership is confirmed next, which matters because each pool needs a clear answer to who controls the cache state during and after the handoff. If the pools use different tensor-parallel layouts, the work may need to be redistributed across their workers rather than copied as one undivided block of bytes. The transfer has failed, and the reserved destination entry is released instead of being left looking like usable cache. The router still retains the request, so responsibility for it hasn’t disappeared; the handoff needs cleanup before a retry, and the request may need its state recomputed if that’s what recovery requires.

68:45 When the disaggregation handoff pays off

Okay, let’s work out the ideal data-movement cost using the same unsharded KV example as before. This multiplication counts two vectors across thirty-two layers, eight thousand one hundred ninety-two tokens, eight KV heads, a width of one hundred twenty-eight numbers per head, and two bytes per number. We’re comparing that payload with an illustrative effective bandwidth of twenty-five billion bytes per second. Multiplying those factors gives a payload of one billion seventy-three million seven hundred forty-one thousand eight hundred twenty-four bytes. That’s the amount we’ll divide by the bandwidth to estimate the transfer time. Now divide the payload in bytes by the bandwidth in bytes per second, and the units leave us with seconds. The calculated duration is about zero point zero four three seconds. That’s about forty-two point nine five milliseconds. It gives us a useful idealized starting point for thinking about the handoff, but we still need to account for what this estimate leaves out.

This is an idealized payload-transfer estimate, not the time for a complete handoff. The number stays the same here; we’re just making clear what kind of estimate it is. And here are overheads the calculation hasn’t counted. Setup and synchronization take extra work, contention means other transfers may compete for bandwidth, and layout conversion may be needed to arrange the cached state for the receiving side. These are comparisons, not extra numbers folded into the estimate. So this forty-two point nine five millisecond estimate isn’t a measured result for either vLLM or SGLang. It comes from the stated payload and illustrative bandwidth, not from timing either engine. The actual payload can change too. Different cache representations or sharding can change how much data must move, and if a prefix is already resident at the destination, less state may need transferring.

The byte calculation gives us one part of the handoff cost, but it doesn’t tell us whether splitting prefill and decode is worthwhile. To see why the answer depends on the workload, let’s compare the complete paths, including waiting, and remember these bands have no numerical time scale. Okay, let’s start with the waiting stages. When the work is colocated, there’s one queue before the work shown here. With separate pools, prefill and decode have their own queues, so we need to follow that path as well. Both paths then include prefill, the work of processing the prompt. In the colocated path it happens alongside decode on the same pool; in the disaggregated path it happens in the separate prefill pool. The disaggregated path also needs a transfer stage, because the prompt’s cached state has to reach the decode side before it can continue. That handoff is the extra work our calculation was estimating. After the transfer, there’s another place to wait, this time in the decode queue. So the handoff cost isn’t the whole story; the waiting on both sides matters too. Decode is ready only at the end of each path, after all its stages. That’s why the sentence here asks us to compare the entire critical path, not just the transfer band or one queue in isolation. In this regime, the colocated queue is long. Separating the pools reduces the waiting enough that queue isolation can outweigh the added handoff. The bands are illustrative, so this shows the relationship, not a measured speedup. Separate pools can also be sized for different needs. If prompt processing and generation need different amounts of capacity, we can provision each pool to suit its own work instead of making one shared pool serve both in the same way. But with light load, the colocated queue is short, while the disaggregated path still includes the transfer and its separate queues. In that situation, colocation may be simpler and faster. So the handoff pays off only when the workload’s queueing and capacity benefits outweigh the extra path it adds.

72:43 Build for useful throughput, not a feature checklist

Those long prompts, short answers, and bursts of waiting requests all put different demands on the engine, so we need a load test that feels like the real service. A load test sends a chosen stream of requests through the system so we can see how it behaves under that work. Okay, let's build that workload from what your application actually sees: how long the prompts and answers are, whether requests reuse the same beginnings, and whether arrivals come steadily or in bursts. Those details change how much processing and memory the engine needs. A stream of identical short requests can make a system look healthy while missing the hard parts of real traffic. The load test is carrying that representative traffic toward the serving engine, so the results depend on what we put into the workload.

Let's turn to the deployment that will run this test. Start with the model and its release, meaning which model version and engine version we're actually evaluating; that gives us a concrete setup to compare. Now the parallel layout is the choice of how the model's work and copies are arranged across the available processors. It has to fit the model, while leaving memory for the live KV state, the saved attention information each active conversation needs as it generates. And let's keep the runtime settings with that deployment too, including the version and hardware. If those change between tests, we cannot tell whether a result came from the one configuration change we meant to compare.

With the baseline recorded, let's change one relevant setting at a time and raise the offered load. Metrics are measurements from the running service, and the feedback path brings those measurements back to the load test so the comparison can guide the next configuration. Here, time to first token is how long a request waits before any answer appears, while inter-token latency is the gap between successive generated tokens. We should also watch the slow end of those waits, not just the average. A high percentile tells us the wait that a chosen fraction of requests stays below, so unusually long delays do not disappear inside an average. Now let's separate the work going in from the work coming out. Prompt tokens are the pieces of the input the model processes; output tokens are the pieces it generates. And errors belong in the record too, including failures and cancellations, because dropping them would make an unhealthy test look better than it is.

When arrivals exceed what the engine can serve, an unbounded queue just keeps growing and makes people wait longer. Bounded admission means limiting how much work enters that queue, so overload is controlled rather than allowed to pile up. The useful comparison now is latency, completed work, and reliability together; the goal is to optimize completed work within your service targets. And that brings the pieces together. vLLM and SGLang offer ways to arrange capacity, reuse work, and schedule requests, but enabling the most features is not the goal. We want the configuration that finishes useful answers while meeting the service's latency and reliability targets, and these measurements let us judge that.