GPU to pixel
Why a chat reply that streams smoothly on a laptop can arrive in production as one late lump, or with ��� where an emoji should be.
Contents7 sections
A streamed AI reply isn’t one pipe from the model to the screen. It passes through a chain of buffers, each counting in its own unit, such as tokens, bytes, events or frames. Most streaming bugs happen at a handoff where one buffer waits to fill, or where the units on each side don’t line up.
A relay, not a pipe
The model writes tokens, which are words or pieces of words, each stored as a number called a token ID. The reply then passes through hops, stages that receive it and pass it on, each with its own buffer.
| Hop | Counts in |
|---|---|
| GPU | token IDs, one per step |
| Server | bytes and server-sent events |
| Compression (gzip) | windows of bytes |
| Proxy | bytes, released when its buffer fills |
| Network | chunks, whatever arrives in one read |
| JavaScript | UTF-16 code units |
| Markdown renderer | blocks, such as a paragraph or a table |
| Screen | graphemes (what you see as one character), in frames about 16.7 ms apart |
Some hops hold the whole reply
A common report is that the stream works on a laptop, but in production the page shows nothing, then the whole answer. The usual first fix is proxy_buffering off in nginx, a proxy that sits in front of your server. Try that in the figure.
The relay
Model of one reply, not a recorded traceWhat the reader sees
A grey line is a token moving down. An orange run is a token held. Change a setting, then press Play.
proxy_buffering is Time runs left to right. Each token falls through 5 lanes, one per hop, and an orange run marks the lane holding it. The model sends 1 token every 0.15 s, and a full proxy buffer releases 6 at a time; both numbers are examples. gzip collects a window of bytes before it writes. This short reply never fills one, so gzip holds every token to the end.
Turning off proxy buffering changes nothing in the chat bubble, because gzip still holds every token. The bubble can’t tell you which hop held the stream, but counting bytes into and out of each hop can.
Any hop that needs a full window, or the whole body, before it sends turns a stream into a download. So turn compression off on streaming endpoints, or flush after each event. Middleware, code that wraps every response on the server, can need the whole body too. In Julia Evans’s “nginx bug”, the culprit was mostly an ETag middleware in Rack, the layer under most Ruby web apps. An ETag is a hash of the whole body used for caching, so it can’t be sent until the reply ends.
The extreme case is a reply that isn’t streamed at all. The AI assistant my team integrated, built by another team in the company, worked that way. In the client library I reviewed, the whole answer came back in one response, and the page then revealed it word by word.
I measured about a minute for one generated page summary and asked for a feasibility assessment. The measurement that came back, from 24 non-production requests, averaged about 21 to 24 seconds, with the slowest close to a minute. I never saw how the assessment ended.
Streaming doesn’t make an answer like that finish sooner. It puts the first words on screen long before the last ones.
A token is not a character
The detokenizer is the server step that turns token IDs back into text. IDs become bytes, bytes become code points (Unicode’s numbered characters), and code points group into graphemes. In UTF-8, the web’s usual encoding, 🦙 is 1 code point and 4 bytes. 👩💻 is 1 grapheme, 3 code points and 11 bytes. Many tokenizers split along bytes, so one emoji can span several tokens.
A decoder prints �, the replacement character, for bytes that don’t form a character. Move the cut.
Byte microscope
Runs in your browserDrag the cut, use the arrows, or tap any byte. The results below change at once.
Decode each chunk alone
Decode with {stream: true}, after chunk 1
after chunk 2
Each column is one grapheme. The cut stands for any boundary that splits bytes, such as a token boundary on the server or a network chunk on the client.
The count of � tells you where the cut fell. A cut after the first byte of 🦙 gives 4.
A server that decodes each token alone sends that � to every user. The fix is to hold tokens back until their bytes form valid UTF-8, as Hugging Face’s DecodeStream does.
A chunk is not a message
Even a server that sends only whole characters, as hosted APIs do, can’t stop the network from cutting the bytes again. In the browser a response body arrives as chunks, and a chunk is whatever bytes one read returned. Locally, one write usually arrives as one chunk, so a broken parser passes every local test.
2 client bugs cause most of the trouble. First, a new TextDecoder per chunk turns a split character into �. LangChain.js’s Vertex adapter shipped this in 2024. Second, server-sent events (SSE), the usual streaming format, let a line end in a carriage return (CR), a line feed (LF) or both (CRLF). A parser that splits on "\n\n" never matches a CRLF stream, so nothing renders.
The fix keeps one decoder per stream and accepts all 3 line endings. It passes a test with 🦙, 👩💻 and 。 in CRLF lines, cut into chunks of every size.
async function* sseData(body) {
// one decoder per stream
const decoder = new TextDecoder();
let buf = "", data = [];
for await (const chunk of body) {
// stream: true keeps a split
// character for the next chunk
buf += decoder.decode(chunk, { stream: true });
let m;
// CR, LF, CRLF all end a line
while ((m = /\r\n|\r|\n/.exec(buf))) {
// a final CR may be
// half of a CRLF
const atEnd = m.index === buf.length - 1;
if (m[0] === "\r" && atEnd) break;
const line = buf.slice(0, m.index);
buf = buf.slice(m.index + m[0].length);
if (line === "") {
if (data.length) yield data.join("\n");
data = [];
} else if (line.startsWith("data:")) {
data.push(line.slice(5).replace(/^ /, ""));
}
}
}
}The browser counts in blocks and frames
Markdown’s unit is the block, such as a paragraph or a table. While a reply streams, the last block is open and can change type. **Menu** is literal asterisks until the closing pair arrives.
Chrome’s guide to streamed replies says to append new text, not re-parse the whole message per token. Re-parsing makes the work grow with the square of the reply’s length. The screen redraws about 60 times a second, so collect the tokens that arrive between redraws and add them to the page in one write.
The one buffer you add on purpose
Vercel’s AI SDK ships smoothStream, which holds text briefly so it comes out at an even pace. It’s a jitter buffer, like the one a video player uses to smooth out uneven arrival. It adds a little delay because text that lands in bursts is harder to read.
The word-by-word reveal in the assistant I reviewed did the same pacing, but only after the whole answer had arrived. It smoothed the text, not the wait.
Where to look first
| What you see | Where it starts | Fix |
|---|---|---|
| � that stays | server decodes each token alone | hold bytes until valid UTF-8 |
| � only in production | client decodes each chunk alone | one TextDecoder, stream: true |
| an empty bubble | parser misses CRLF | accept CR, LF and CRLF |
| nothing, then everything | gzip or body-hashing middleware | flush after each event |
| text in bursts | proxy buffer | turn proxy buffering off |
| raw asterisks | Markdown block still open | parse per block |
| slower as it grows | re-parse on every token | append |
| stutter | one page write per token | one write per frame |
The one test I’d add to any streaming endpoint runs through the production proxy. It sends emoji and CRLF line endings cut at random byte offsets, and times the first word at the client.