13 July 2026

I taught a locked text-to-speech model to speak in a voice that doesn’t exist

I’ve been building a voice assistant that runs entirely on my laptop. No cloud, no API calls, everything on an Apple M4. It hears you, reasons about your code, and talks back in its own voice. This is the story of how it got that voice, which turned out to be a much stranger problem than I expected.

The voice I couldn’t have

The assistant speaks through Kyutai’s open text-to-speech model, which is genuinely excellent and runs fast enough to hold a conversation. It ships with a library of voices, and for a while I just used a blend of two of them. Then I decided I wanted something specific: a calm Australian voice. I’m Australian. I have to listen to this thing every day. Seemed reasonable.

It is not, it turns out, on the shelf. The model’s voice library has exactly two Australians in it and they are both men. The nearest woman is a New Zealand speaker, and I’m sorry, but I am not building my daily assistant around a Kiwi accent. So I went looking for how to make a new voice from scratch.

Here’s the catch. The model conditions its voice on a small tensor, and that tensor is produced by a separate encoder that Kyutai never released. The public checkpoints give you pre-made voices and no way to make your own. Someone had already filed the issue and confirmed it: the voice encoder is missing, and without it, you get whatever voices they decided to ship. Voice cloning into this model is closed by design.

I found this out at about the point where a sensible person stops. But the tensor is just numbers, and numbers are fair game.

Picking the lock

The voice tensor is 64 thousand floating point values, fed into the model through a single linear layer. The encoder that would normally produce it is gone. But I don’t actually need the encoder. I need a tensor the frozen model responds to correctly, and there is more than one way to find one.

So I froze the entire model, every weight locked, and made that voice tensor the only thing that could change. Then I ran gradient descent on it. Given a few minutes of a target voice with transcripts, I minimised the model’s own teacher-forced loss: how surprised is the model by this audio, if I tell it the voice is this tensor? Tune the tensor until the model stops being surprised. No encoder required. Just audio, transcripts, and the model’s own gradients pointing the way.

If this sounds familiar, it’s textual inversion, the trick from image diffusion models where you learn a new embedding for a concept the model was never trained on. Nobody, as far as I can find, had pointed it at voice conditioning for a speech model. It’s a small idea. Most good ones are.

The beautiful part is that I could check it honestly, because I already had a voice whose real tensor ships with the model. I inverted a voice I had ground truth for, and compared. The optimised tensor scored better loss than the real one, and more importantly, it sounded like the same person. To my ear, with my own known voice as the reference, it was her. That was the “wild” moment. The lock had a key, and I could cut it.

A voice that belongs to nobody

Once you can invert one voice, something better opens up. The loss doesn’t care whether all your audio comes from one person. Feed it several speakers at once and it will find the single tensor that best explains all of them. That tensor is a kind of centroid: a voice that is the average of the inputs and the recording of nobody.

I used this on purpose. The final voice is inverted from a professional audiobook narrator (warm register, excellent timing, public domain) blended in the training mix with clips from consenting Australian speakers in an open crowd-sourced dataset (accent, inflection, the feel of the thing). The ratio is a dial. Quality comes from the narrator, character comes from the Australians, and the result is a voice that no real person has ever spoken with. Every source is public domain or CC0. It sidesteps the whole ethical mess of cloning a specific human, and the assistant ends up with a voice that is genuinely its own.

The part where it started speaking in tongues

I nearly shipped it too early. The voice sounded great across a handful of test sentences, so I moved on. Then I generated ten thousand sentences of training data and ran my speech recogniser over all of them to check quality, and about one in five came back as complete gibberish.

Not noise. Gibberish. Perfectly pronounced, confident, fluent nonsense. Beautifully articulated syllables assembled into no words at all, like someone speaking a language that doesn’t exist. Some samples even came out in Chinese characters. The voice was flawless. It just occasionally had nothing to say and said it anyway.

The reason took a while to accept, because it broke my mental model. My optimised voice tensors had excellent loss, better than several of the shipped voices. But loss did not predict this failure at all. The inverted tensors sit slightly off the manifold the original encoder would have produced, in a direction the loss function is completely blind to. And the failure is bistable: each generation either locks onto the text in the first few frames and stays perfect, or misses and then commits, gorgeously, to babble.

I now treat this as a law of the project. You cannot judge one of these voices on five samples. You generate a couple of hundred, score them automatically, and read the actual failure rate. My five-for-five test had just been lucky.

Fixing it two ways

For training data, the fix is a loop: score everything, throw out the gibberish, regenerate it with fresh randomness, keep the takes that pass, repeat. That drives the failure rate to zero at the cost of a bit of extra compute. The models check each other’s work, which is a theme in this project. The code model writes the sentences, the TTS speaks them, the recogniser audits the TTS, and the recogniser decides what’s allowed to become training data.

For the live assistant, throwing away bad takes isn’t an option, so I needed the voice itself to stop wandering. Pulling the tensor a short way back toward the safe region of shipped voices did it. Fifteen percent of the way back killed the gibberish entirely, and to my ear it cost nothing: the voice still sounds like itself. There’s a nice self-distillation trick in here too, where I re-invert the voice using the model’s own cleanest generations, since those come with perfect alignments. The voice teaches itself to behave using its own best behaviour.

Why bother doing this on a laptop

All of this runs on device. That isn’t a purity thing, it’s the actual bet behind the project. A voice assistant that reasons about your private code should not be shipping that code to someone else’s servers, and it turns out you don’t have to. The whole pipeline, the codec written from the papers, the models running through Apple’s MLX, even an 8-billion-parameter teacher model I had to re-quantise myself because the only public version was numerically broken, fits on one machine.

The voice was supposed to be a footnote. Pick a nice one, move on. Instead it became a week of gradient descent against a system that was specifically built not to let me do this, and it ended with a calm, warm, faintly Australian voice that exists nowhere except as the solution to an optimisation problem.

I think that’s the most honest way to build a voice for a machine anyway. Not stolen from a person, not picked off a shelf. Solved for.