Written 13 July 2026 · published 15 September 2026

I taught a locked text-to-speech model to speak in a voice that doesn’t exist

I’m building a voice assistant that runs entirely on my laptop: no cloud, no API calls, everything on an Apple M4. It hears you, reasons about your code, and talks back in its own voice. This is how it got that voice.

The voice I couldn’t have

The assistant speaks through Kyutai’s open text-to-speech model. It is very good and fast enough to hold a conversation. It ships with a library of voices, and for a while I used a blend of two of them. Then I wanted something specific: a calm Australian voice. I’m Australian and I have to listen to this thing every day.

The library has two Australians and both are men. The nearest woman is a New Zealand speaker, and I am not building my daily assistant around a Kiwi accent. So I looked into making a new voice from scratch.

The model conditions its voice on a small tensor. That tensor comes from a separate encoder that Kyutai never released. The public checkpoints give you pre-made voices and no way to make your own; someone had already filed the issue and had it confirmed. Voice cloning into this model is closed by design.

That is the point where a sensible person stops. But the tensor is just numbers, and numbers can be optimised.

Picking the lock

The voice tensor is 64 thousand floating point values, fed into the model through one linear layer. The encoder that produces it is gone, but I don’t need the encoder. I need a tensor the frozen model responds to correctly, and there is more than one way to find one.

I froze the entire model, made the voice tensor the only thing that could change, and ran gradient descent on it. Given a few minutes of a target voice with transcripts, I minimised the model’s own teacher-forced loss: how surprised is the model by this audio if I tell it the voice is this tensor? Tune the tensor until the model stops being surprised. It needs no encoder, only audio, transcripts and the model’s gradients.

This is textual inversion, the technique from image diffusion models where you learn a new embedding for a concept the model was never trained on. As far as I can find, nobody had applied it to the voice conditioning of a speech model.

Before optimising anything I checked that the loss could tell voices apart. On held-out speech from one voice, the model’s surprise was 5.28 nats with that voice’s real tensor, 6.23 with the average of four other shipped voices, and 7.24 with noise. Nearly a full nat separates the right key from a plausible wrong one, so the surface slopes toward the truth.

I could check the method properly too, because that shipped voice comes with its real tensor. Starting from the wrong-voice average, held-out loss fell from 6.35 to 2.97, past the real tensor’s own 5.28, because the optimised tensor is tailored to this speaker and these recordings where the shipped one is a general prior. The recovered voice transcribed word-perfect under my own speech recogniser, and it sounded like the same person.

A voice that belongs to nobody

Once you can invert one voice, you can invert several at once. The loss doesn’t care whether the audio comes from one person. Feed it several speakers and it finds the single tensor that best explains all of them: an average voice, the recording of nobody.

I did this on purpose. The final voice is inverted from a professional audiobook narrator (warm register, good timing, public domain), blended in the training mix with clips from consenting Australian speakers in an open crowd-sourced dataset (the accent and the inflection). The ratio is a dial. The quality comes from the narrator, the character from the Australians, and the result is a voice no real person has spoken with. Every source is public domain or CC0, so nobody’s voice was cloned.

The part where it started speaking in tongues

I nearly shipped it too early. The voice sounded good on a handful of test sentences, so I moved on. Then I generated ten thousand sentences of training data, ran my speech recogniser over all of them, and about one in five came back as gibberish.

Not noise. Perfectly pronounced, fluent nonsense: well-formed syllables that made no words, and some samples came out in Chinese characters. The voice was fine. It occasionally had nothing to say and said it anyway.

The reason took a while to accept. My optimised tensors had excellent loss, better than several of the shipped voices, and loss did not predict this failure at all. The shipped voices babble on roughly one generation in a thousand; every tensor I optimised babbled on ten to twenty in a hundred. The inverted tensors sit slightly off the manifold the original encoder would have produced, in a direction the loss function can’t see. The failure is bistable: each generation either locks onto the text in the first few frames and stays correct, or misses and commits to babble.

The rule I took from this: you cannot judge one of these voices on five samples. Generate a couple of hundred, score them automatically, and read the failure rate. My five-for-five test had been lucky.

Fixing it two ways

For training data, the fix is a loop: score everything, throw out the gibberish, regenerate it with fresh randomness, keep the takes that pass, repeat. That drives the failure rate to zero for a little extra compute. The code model writes the sentences, the TTS speaks them, and the recogniser decides what is allowed to become training data.

For the live assistant, throwing away bad takes isn’t an option, so the voice itself had to stop wandering. The most principled fix failed in the most informative way. I re-inverted the voice from 7,374 of the model’s own cleanest generations, whose word alignments are exact because the model recorded them as it spoke. That gave the best held-out loss of any attempt, 2.84, and the worst babble: 14.5% of generations severe. A tensor can make the target audio more likely while making everything else, the model’s babble included, more confident. Teacher-forced likelihood does not measure whether a voice will hold onto the words.

What worked was blunter. Pulling that tensor fifteen percent of the way back toward the average of the shipped voices took severe babble from 14.5% to zero across the same test set. Thirty percent bought nothing further. When I listen, the voice still sounds like itself.

Why it runs on a laptop

All of this runs on device. That is the bet behind the project: a voice assistant that reasons about your private code should not send that code to someone else’s servers, and it doesn’t have to. The whole pipeline fits on one machine: the codec, written from the papers; the models, running through Apple’s MLX; and an 8-billion-parameter teacher model I re-quantised myself because the only public version was numerically broken.

The voice was supposed to be a footnote. It took a week of gradient descent against a system built not to allow it, and ended with a calm, faintly Australian voice that exists only as the solution to an optimisation problem.