Written 28 July 2026 · published 18 September 2026
From to tokens a second without touching a matrix multiply
The assistant’s brain decoded at 22.8 tokens a second, when the laptop’s memory allows about four times that. I put the loss down to the hundreds of small GPU launches a token needs, and built speculative decoding to spread them over several tokens at once. It missed the bar I had set for it, and the profiler written to plan the next attempt showed the launches cost almost nothing: a normalisation kernel from the project’s first week, running on a single GPU thread, was taking forty percent of every token. Forty lines of Metal later, the brain decodes at 42.8. This follows the continuous thread.
The strip is one token of that brain, measured again the day this went out, with each kernel in turn. The chapters set its window, from a second down to one launch. Press either number under it and the essay is produced a token at a time at that kernel’s rate.
The plan, and the number it had to reach
A language model produces one token at a time, and each token costs a full pass through the model. Speculative decoding tries to get several tokens for the price of one: a small draft model guesses the next few cheaply, and the big model checks all the guesses in a single pass, which is only worthwhile if the guesses are usually right and checking several at once costs about the same as checking one. My draft was a 59-million-parameter head reading the vector the brain computes for each token before it chooses a word, so it needed no model of its own. Before building it I wrote down the number it had to reach to ship: one and a half times faster, end to end. Under that, the work would be kept and the campaign closed.
The build went the whole way: dumps of the brain’s per-token vectors to train on, a trainer for draft heads of increasing depth, a Rust port checked against the Python to the bit, and a serving loop that, with speculation switched off, reproduces plain decoding exactly. It died on two numbers.
The draft head plateaued. Its accuracy on the second guessed token, on text it had not seen, stopped at 0.405 against the 0.5 the method needs, which works out to about two accepted tokens per cycle where the method wants three or more, and four thousand more training steps did not move it. And checking several tokens at once was not nearly free. The multiply for more than one token works in blocks of sixty-four, so checking two tokens costs the same 363 milliseconds as checking sixty-four, against 43 milliseconds for one plain step. With every fix granted in advance, the ceiling came out between 1.1 and 1.45 times, under the line. The loop stays in the tree, bit-exact, in case a future draft head earns it. Killing a campaign against a criterion written before it started feels clean in a way abandoning one never does.
The handles are that cycle: the check, the cost of a draft, the tokens kept. They open at the loop as measured today, where the check is cheap (a small-batch kernel written since) and the cycle still loses.
What one token costs
The post-mortem still pointed at the launches. Each token needs hundreds of separate pieces of GPU work, and the next plan was one large kernel that would do them all without returning to the CPU between them. Its first step was to measure the launches: what the CPU spent issuing work, what the GPU spent running it, from the GPU’s own timestamps, and each kind of kernel on its own.
On the base M4, quiet, one token took 43.9 milliseconds and 291 launches, not the five hundred that had been assumed. The CPU spent 5.6 milliseconds issuing them, so the launches were not the cost. The GPU spent 38.2 milliseconds running, but reading memory at an effective 50 gigabytes a second, when every heavy kernel timed on its own reached 95 to 126. Work that had to wait for earlier work cost nothing extra; the GPU had the next piece queued. Eighteen milliseconds a token were going somewhere else, and the per-kernel timings found it: the normalisation that keeps each layer’s numbers in range, a kernel that should take two microseconds on a row of 2048 values, took 231.
One thread
That kernel had been written in the project’s first week, when the point was to prove that the whole thing could run at all. It used one GPU thread per row, walking the 2048 values one after another while the chip’s other cores sat idle. It was correct, and it was invisible while the matrix multiplies around it were slow. Every optimisation since (caching the compiled pipelines, one command buffer per token, fusing the projections, flash attention) made everything around it faster, until one thread on one core was forty percent of every token. It runs twice per layer and once at the end, 73 times a token. The other normalisation in the file, the layer norm, had the identical fault, and it sits on the speech model’s per-frame path and in the codec.
The fix is a parallel reduction:256 threads share each row,each sums its part,and the parts are combined across the group,in two passes for the layer normso its results matchthe serial version’sto the last bit that matters.
About forty lines of Metal for each. The norm went from 231 microseconds to 2.7. Decode went from 22.8 to 42.8 tokens a second, 23.4 milliseconds a token, and the GPU’s effective memory rate from 50 to 90 gigabytes a second. The speech path inherited the layer-norm fix and renders a frame in 36 of its 80 milliseconds. Every check still passes, the speculative loop bit for bit. The order of the additions changed, so the lowest bits of the output drift, and that was checked to be harmless.
With WebGPU, the corner control runs this kernel on your GPU at every thread count from one to 256, checking each result before it reports a time. Your numbers will sit above the essay’s, since a WebGPU launch costs more than a Metal one, with the same shape.
The kernel that was never built
There is a floor under all of this. A token cannot take less time than reading every weight in the model once, and at the memory bandwidth the kernels have already shown, that is 18.5 milliseconds. After the fix the GPU’s time per token is 21.3. Everything the large kernel could still recover, the gaps between pieces of work and two milliseconds of CPU, is about fifteen percent, and it is the riskiest kernel work on the menu. It is filed for later.
I have learned this three times now, so it is a rule: any kernel that can end up on a single thread gets a cooperative version before it ships, and the boring operations get profiled first. The two slowest kernels in the system were the two simplest in the file, the ones I never benchmarked because there was obviously nothing there to measure.
The corner control measures your device’s memory bandwidth and puts a token of this brain on the ruler at that rate, beside the floor and the measured token.