Raphael Friedmann
← Understanding AI

How a model hears and speaks

Dictation, meeting transcripts, voice assistants and cloned voices all rest on one idea: turn sound into a picture, then into tokens. How speech recognition and speech synthesis work, where they fail, and what to watch for at work.

A voice assistant, as the lesson on AI that isn’t an LLM put it, is usually three models in a row: one turns your speech into text, an LLM decides what to say, and a third turns the answer back into speech. This lesson opens the first and the last of those boxes. Both turn out to use ideas you have already met: the vision approach of turning a picture into vectors, and the next-token guessing of a language model. That also explains their typical mistakes.

Sound is a very long list of numbers#

A microphone measures air pressure, many times a second. Each measurement is one number, called a sample. A music CD stores 44,100 of them per second; speech models usually make do with 16,000, which is what Whisper, the best-known open speech recogniser, works with. Thirty seconds of speech is therefore 30 x 16,000 = 480,000 numbers.

That is far too many, and far too raw. A single number says nothing about whether you just heard an “a” or an “s”. What matters for speech is which pitches are present and how strongly, and how that changes over time. So before any model sees the audio, it is turned into a picture.

Whisper slices the audio into windows of 25 milliseconds, one every 10 milliseconds, and measures 80 pitch bands in each (128 since large-v3). That gives 100 columns per second, so its standard 30-second chunk becomes a picture of 3,000 columns by 80 rows. From here on, hearing is a vision problem.

From sound wave to text: hearing is reading a picture
sound wave16,000 samples/sspectrogram100 columns/sencoderlistenslike a vision encoderdecoderwrites wordsa small language modeltextthe decoder guesses the next word like any LLM,which is why it can invent text where there is only silence
The last column of the spectrogram is silence: nothing lit. The encoder turns the picture into vectors, the decoder writes the transcript one token at a time, looking back at those vectors. Same encoder-decoder shape as in machine translation, with sound as the source language.

Hearing: an encoder that listens, a decoder that writes#

Whisper, published by OpenAI in 2022 (Radford et al., “Robust Speech Recognition via Large-Scale Weak Supervision”), is the model most people meet first, because its weights are open under the MIT licence. It has two halves. The encoder reads the spectrogram and turns it into vectors, much like the vision encoder turns image patches into vectors. The decoder then writes the transcript token by token, exactly like a language model, except that at every step it also looks at what the encoder heard.

What made Whisper good was not a novel architecture but data: 680,000 hours of audio with transcripts collected from the web. “Weakly supervised” means those transcripts were imperfect, subtitles and captions of mixed quality, but there was so much of it that the model learned accents, background noise and dozens of languages anyway.

Why it writes things nobody said#

The decoder is a small language model, and language models fill gaps with something plausible. Give it a long pause, music or a cough, and it may write a sentence that was never spoken. A 2024 study (“Careless Whisper”, Koenecke et al., FAccT 2024) found invented phrases in about 1% of transcriptions, and 38% of those inventions contained something harmful, such as violence or made-up personal details. The tests ran on the 2023 version; newer models do better, but the mechanism is still there. Speakers with long pauses were hit harder.

Two practical consequences. Most transcription tools have an option to detect and cut silence before the model sees it (voice activity detection): switch it on. And a transcript is a draft, not a record, especially for anything that goes into minutes, a ticket or a medical note.

The standard quality measure is the word error rate: count the words the model got wrong, left out or added, and divide by the number of words actually spoken. A word error rate of 5% sounds good and still means one word in twenty is wrong.

Speaking: the same trip in reverse#

Text-to-speech runs the pipeline backwards. The older approach: a model turns text into a spectrogram, and a second model, the vocoder, turns that picture back into a sound wave. The newer approach borrows from language models directly. A neural audio codec (Meta’s EnCodec from 2022 is a well-known one) compresses audio into a short sequence of discrete tokens, a vocabulary of sound snippets. A model then predicts those audio tokens one after another, exactly as an LLM predicts words, and the codec turns them back into audio.

That second approach is what made voice cloning cheap. Microsoft’s VALL-E (January 2023) was trained this way and could imitate a speaker it had never heard from a three-second recording, by using that recording as the start of the token sequence and simply continuing in the same voice.

Cloned voices, real money#

Three seconds of audio is a voicemail greeting, a conference talk, a podcast. In January 2024 an employee of the engineering firm Arup in Hong Kong transferred HK$200 million, about US$25 million, in 15 transfers after a video call in which the “chief financial officer” and the other colleagues were generated, faces and voices.

The defence is not technical, it is a process: payment and password requests by phone or video get a call-back on a number you already have, and some teams agree on a code word. The AI Act also requires that generated or manipulated audio presented as real is disclosed as such (Art. 50); the lesson on GDPR and the AI Act has the dates. Fraudsters will not comply, so the call-back rule stays.

Without text in between#

The three-box pipeline has a cost: every step adds delay, and the LLM in the middle only ever sees text, so tone, hesitation and laughter are lost. Newer models are speech-native: audio in, audio out, one network. OpenAI’s GPT-4o (May 2024) answered spoken input in 320 milliseconds on average, against 2.8 to 5.4 seconds for the earlier three-model voice mode. Kyutai’s open model Moshi (September 2024) listens and speaks at the same time and gives a theoretical delay of 160 milliseconds; the average gap between speakers in human conversation is about 230.

Three models in a row

  • Every step is inspectable: you can read the text
  • Swap one part, keep the rest
  • Runs well locally on modest hardware
  • Slower, and tone gets lost in the middle

One speech-native model

  • Answers about as fast as a person
  • Hears tone, can be interrupted
  • Harder to log and check what was "said"
  • Mostly cloud services today

Running it on your own machine#

Speech recognition is one of the easiest things to run locally. The current Whisper large-v3-turbo (October 2024) has 809 million parameters, small next to a chat model: OpenAI cut the decoder from 32 layers to 4 and kept the encoder. Two popular runtimes: whisper.cpp, a C/C++ port that runs on laptop CPUs and Apple machines, and faster-whisper, which its authors put at up to four times faster than OpenAI’s own code on the same GPU. For a meeting recording, local transcription also means the audio never leaves the building.

In the office: before you press record#

A recording of a meeting is personal data, and so is the transcript: voices are identifiable, and what people say in meetings usually is too. In Germany there is a second, older rule on top of the GDPR: § 201 of the Criminal Code makes it an offence to record the non-public spoken word of others without authorisation. Whether a live transcript that never stores the audio counts as a recording is disputed among lawyers. The safe practice is the simple one: announce it, get everyone’s consent, say where the transcript goes and when it is deleted. In a company with a works council, a transcription tool needs its agreement first (§ 87(1) no. 6 BetrVG).

The takeaway#

Hearing is reading a picture of sound with a vision-style encoder; speaking is a language model that predicts sound tokens instead of words. Both inherit the habits of their parts: the transcriber can invent sentences in silence, and the speaker can sound like anyone after three seconds of listening. Use transcripts as drafts, cut the silence, ask before recording, and verify any voice that asks for money.