Rendering speech with 36 million parameters instead of 600 million, and it comes out clearer

The serial pass

The assistant speaks through a neural codec. Audio is a grid of discrete tokens, sixteen codebooks deep, twelve and a half frames a second. To talk, a 1.6-billion-parameter transformer predicts the first codebook for each frame, and a second transformer, the depth model, 600M parameters, predicts the remaining fifteen one at a time. On a laptop whose limit is memory bandwidth, that serial pass is the most expensive thing in the system.

My first attempt at removing it was compression: tokens at syllable rate and a learned expander back to frame rate. Every version warbled. Measured against the reference, the pitch track wobbled around the right melody at a few hertz, the band people hear as vibrato. The expander was rebuilding the pitch contour from too few samples, and no amount of training moved that.

The project’s other failures sat at quantizers too. The warble was information a discrete bottleneck could not carry. The gibberish in the previous post was discrete autoregression committing early to the wrong thing. The one clean success, the voice inversion, had come from optimising a continuous tensor against a frozen model. The published work on low frame rate speech says the same: below ten frames a second, the systems that work decode from something continuous. So the plan changed to predicting the codec’s internal continuous representation directly and leaving the quantizer’s bits on the table.

Measure first

The idea rests on one question: will the codec’s decoder accept a continuous vector it has only ever seen quantized? I fed it unquantized latents from real audio. The output was cleaner than the production path, at the quality of using twice as many codebooks, with zero failures across several hundred machine-scored generations. It also tolerated corruption: latents with forty percent Gaussian noise added were close to indistinguishable from clean ones, by score and when I listened. The target I wanted a model to predict sits in the middle of a wide basin.

The codec wraps its quantizer in learned projections, so the obvious continuous latent lives in a different coordinate system from the one the decoder reads. Skipping the quantizer means keeping the projections and dropping only the discretisation. Done the other way the decoder produces garbage, and the idea would have failed its first test for the wrong reason.

Three rules from a model that didn’t work

The first head was trained from scratch on ten hours of the assistant’s own generated speech. It failed three ways, each with a cause I could isolate.

Conditioning must modulate. The first head was a flow-matching model, one that learns to walk noise towards the target latent in a few steps, and it took its conditioning as one more input. Its loss was respectable and its output was the unconditional average of speech; ablating the conditioning changed nothing. Letting the conditioning scale and shift every layer, as image diffusion models do, took the same data and the same size from that to perfect memorisation.

Corruption must be announced. Training with noised history, so the model would tolerate its own mistakes, produced blur. The model could not tell corrupted input from clean, so it hedged on both. Given the noise level as an input, it stays sharp on clean input and cautious on noisy input.

Text must lead audio. The teacher reads text more than a second ahead of speaking it. My data placed each word at the moment of its audio. The model memorised the training set and could not speak one unseen sentence, because the task as I had built it requires pronouncing a word at the instant it first appears. Shifting the text ahead fixed it.

With all three fixed the model learned, and then hit the ceiling of what ten hours of data can teach. Intelligibility on unseen sentences rose and flattened well below usable. Distilling the teacher’s soft predictions bought one more increment and flattened too. On a laptop there is no training past that with more data.

Rendering from a frozen model’s state

The expensive knowledge was already running. The teacher’s 1.6B backbone reads text well, so I left it frozen, tapped its per-frame hidden state, the vector it hands to the depth model, and trained a small head to decode that into the continuous latent. The backbone does the understanding and the head only has to render. The depth model still runs, because the backbone expects its codes fed back each frame, but what it produces no longer reaches the decoder.

A 26M-parameter head, trained for fifteen minutes on four hundred samples, decoded unseen sentences at the accuracy ceiling of the measurement. Everything the 600M depth model computes, every frame, is recoverable from the backbone’s state by a model one twentieth its size.

Fifteen hypotheses

Every test so far had handed the head the true previous frames as it went, which is called teacher forcing, and under it the head was perfect. Generating for real, it has to feed on its own output. Left to do that, it collapsed within a few frames into glossolalia, in a voice gone hoarse. I ran fifteen controlled experiments over two days, each designed to kill one hypothesis in under an hour: sampling temperature, the noise schedule, feedback fidelity, frozen noise, partial teacher feedback. Each was ruled out in turn.

The cause was the harness. The reference implementation applies classifier-free guidance at generation time, a second forward pass with the text blanked, which keeps the model on the words, plus warm-up and pacing steps. My generation loop had none of them, and I had validated that loop only with the true frames handed in, where none of them matter. A generation harness is untested until it has run without the answers.

Inside the reference generator the head worked as measured. Across unseen sentences, generating live, the teacher’s own decode scores 0.846 with one failure; the head scores 0.859 with none. The student out-decodes the teacher, which is what the first afternoon predicted: continuous latents clear the ceiling that quantization imposes.

Hearing a fault, then measuring it

What remained was texture, and for texture the best instrument I have is listening. The project’s rule is numbers before opinions, but a number only counts when it measures the fault, and I can hear a fault long before I have a number for it. So the order of work was: hear it, name it, build the measurement that reproduces it, then find the cause.

Warble: the pitch track, compared with the reference, wobbled in the 3 to 10 Hz band. A fan-like hiss: the harmonics-to-noise ratio was twelve decibels below the reference, which is the model rendering uncertainty as noise. Unnatural pauses: half were in the teacher’s own pacing and half were the head under-rendering quiet transition frames, which the frame-by-frame level trace showed directly. Six times a metric and my listening disagreed, and each time listening was right; in one case a metric improved while the audio got worse, because the metric rewarded blur.

The fix for texture was adversarial training: a discriminator on the decoded waveform through the frozen codec, hinge loss, the usual stabilisers from image GANs, and a recent method that cuts sampling from eight steps to two while raising fidelity. One four-hour run gave the lowest pitch wobble measured on the project, halved the noise deficit, held intelligibility, and made sampling four times cheaper. It also sounds better.

Production

Porting to the Rust stack took an evening. The head is small enough to run on the CPU beside the text machinery; the codec gained a second entrance that accepts latents instead of tokens; a golden test pins the Rust maths to the Python maths. First generation ran in real time. The quality gate matched the research rig to the third decimal.

Then I talked to it, and the felt latency was five seconds. That number decomposed into three causes: an over-cautious end-of-turn detector, a playback buffer holding more margin than it needed, and the speech system reloading its whole model every turn, which no per-sentence metric could see. All three are fixed. The buffer tunes itself, raising its hold if audio runs thin and giving the margin back when it is flush. And she says “mm-hm” the moment you stop talking, because people don’t answer into silence.

Tokens for deciding, vectors for rendering

Five days end to end, and the decoder path no longer contains the most expensive component in the system. The discrete and the continuous want different work. Every place I made the machine commit to symbols where it should have kept a continuum, it warbled, babbled or blurred: the syllable-rate tokens, the gibberish, the quantized feedback. Every place I let it stay continuous until the last possible moment, it worked: the inverted voice tensor, the unquantized latent, the head that renders it.

Postscript, three days later

The adversarial training went three rounds. Round two targeted the unnatural pauses and recovered them on the meter at the cost of a little polish. Round three balanced the two and produced the first checkpoint in the campaign to win every instrument and the listening test at once. That head now speaks for the assistant.

One decision is worth recording. Cutting the depth model’s remaining stages from eight to three measured better on the intelligibility gate and 28% faster, and I held it back, because I could hear a loss of texture that the word recogniser cannot detect. Two days later, with the sharper round-three head rendering, the same A/B was indistinguishable and the speed-up shipped. Both calls were right: the gate protected quality the old head couldn’t afford to lose, and the better head earned it back. Since the end of July a kernel fix has made the eight-book feedback affordable again, and that is what runs. Seven times now I have overruled a metric by listening and been proved right afterwards, so the project’s rule is fixed: automated gates set floors, and listening decides what ships.

Somewhere in there the machine also, unprompted, prefixed a sentence with “Lemons,” and pronounced “it’s” as “it CSS”. The training corpus is developer chatter, and apparently she dreams in stylesheets. Both takes are preserved. Some artefacts you fix; some you frame.

The conversation is fast now: about two seconds from the end of your sentence to the start of hers. The next block is the brain. Profiling shows the language model spends three-quarters of every token’s time on dispatch overhead, not memory bandwidth, which rules out textbook speculative decoding (a draft model pays the same tax) and points at draft heads on the model’s own hidden state, trained with the same frozen-backbone pipeline that built the speech head. It is the same trick a third time, on a different organ.