How Text-to-Speech Works
Two stages: text becomes a spectrogram, the spectrogram becomes audio. Plus why prosody is the hard part and latency is the practical one.
On this page
Text-to-speech runs speech recognition backwards, and it splits into two stages that are usually separate models.
Stage one turns text into a mel spectrogram — the same frequency-over-time picture that recognition consumes.
Stage two, called a vocoder, turns that spectrogram into an actual waveform.
The split exists because the two problems are different in kind. Predicting what the sound should look like is a language and prosody problem. Producing convincing audio samples from that description is a signal generation problem.
Stage one: text to spectrogram
Input text is normalized first, and this stage causes more real-world errors than the neural parts. Dr. might be “doctor” or “drive”. 1996 might be a year or a quantity. St. might be “street” or “saint”. Getting these wrong produces confidently mispronounced output, and the fix is rules and dictionaries rather than a better model.
Text — or a phoneme sequence derived from it — is then encoded and mapped to spectrogram frames.
The essential difficulty here is that the mapping is one-to-many. A sentence has no single correct rendering. Which words get stressed, where pauses fall, whether the pitch rises at the end — all are valid variations, and the model must choose one. Averaging over possibilities produces the flat, characterless delivery that older systems were known for.
Prosody is the whole game, and it depends on meaning. “I never said she stole it” has seven distinct readings depending on stress. A model that does not represent intent cannot reliably pick the right one.
Stage two: spectrogram to waveform
A spectrogram discards phase information, so reconstruction is not a simple inversion — the vocoder must generate plausible waveform detail.
Early neural vocoders generated audio sample by sample, autoregressively. Extremely high quality, and far too slow for practical use at tens of thousands of samples per second.
Current systems generate in parallel instead, typically adversarially trained: a generator produces audio, a discriminator judges whether it sounds real, and they train against each other. Quality approaches the autoregressive approach at a small fraction of the cost, which is what made real-time synthesis practical.
Voice cloning
A speaker embedding conditions the synthesis, capturing timbre and speaking style as a vector.
With enough recorded audio from one speaker, you can fine-tune a highly faithful voice. More striking is zero-shot cloning: a few seconds of reference audio produces a speaker embedding good enough for recognizable imitation, with no training at all.
This is genuinely useful — accessibility, localization, consistent brand voice — and genuinely dangerous. Voice is used as an identity signal in banking, in family relationships, and in journalism. Two things follow that are worth stating plainly: obtain consent before cloning anyone’s voice, and treat voice as no longer sufficient for authentication. Voice-based verification of identity is not reliable, and building systems that depend on it is now a design error.
The end-to-end shift
Newer systems reduce or eliminate the two-stage split, generating audio tokens directly from text with a single transformer — treating audio as a sequence of discrete tokens from a learned codebook, exactly as text uses a vocabulary.
The advantage is prosody. A single model that sees text and produces audio tokens can condition intonation on meaning across the whole utterance, rather than committing to a spectrogram before knowing how it will sound. This is where quality gains have come from recently.
Practical notes
Latency splits into two numbers, as with streaming text: time to first audio, and generation speed relative to real time. For conversational applications the first matters far more — a delay before speech starts is what feels broken.
Stream both stages. Synthesize sentence by sentence as text arrives from a language model rather than waiting for the full response. This is the difference between an assistant that feels responsive and one that does not.
Control markup varies. Most systems accept some way to specify pauses, emphasis, or pronunciation. Worth learning for any production use — it is how you fix the cases the model gets wrong.
Test on your actual text. Numbers, abbreviations, code, and proper nouns are where synthesis embarrasses itself, and generic demos never include them.
What to remember
- Two stages: text → spectrogram (prosody problem) and spectrogram → waveform (vocoder, signal problem).
- Text normalization causes more errors than the neural components — abbreviations and numbers especially.
- The mapping is one-to-many; prosody depends on meaning, which is why end-to-end models are improving it.
- Parallel adversarial vocoders replaced sample-by-sample generation and made real-time practical.
- Zero-shot voice cloning needs seconds of audio: get consent, and stop treating voice as authentication.