Neural voice cloning explained: the four stages from sample to speech

October 28, 2026 · 8 min read · By Resona Editorial Team · Resona Blog

Almost every article about voice cloning describes what it feels like to use: upload a clip, type text, hear yourself say something you never said. Fewer explain the machine. This one does — the four stages that sit between a 30-second recording and a finished audio file, and why each of them exists.

It helps to start with what the word neural is actually contrasting against, because the older approaches are still running in plenty of systems you use every day.

1. Three generations of speech synthesis

Concatenative synthesis (roughly 1990s to 2000s) recorded a single voice actor reading thousands of sentences, cut the audio into tiny units — phonemes, diphones, sometimes whole syllables — and played the right units back in the right order. It sounded convincingly human for the fragments it owned, because those fragments came from a real person. But it needed 10 to 40 hours of studio recording per voice, and the joins between units were audible. You could not clone a new voice at all: you had to book the studio.

Statistical parametric synthesis solved the storage problem by modelling speech instead of storing it. It predicted acoustic parameters from text using hidden Markov models or early neural nets, then a vocoder rebuilt audio from those parameters. Voice identity became a set of numbers you could swap — a real step toward cloning. The catch was texture: parametric output had a characteristic muffled, slightly buzzy quality, because the parameters threw away fine detail that perception notices.

Neural synthesis removed the trade-off. Deep networks learn the mapping from text to audio directly, and a second network — the vocoder — generates the waveform sample by sample rather than assembling or approximating it. Texture came back. And crucially, the same architecture turned out to be very good at separating what is said from who is saying it, which is exactly the operation cloning needs.

That separation is the whole trick, so the rest of this article is really about how it is implemented.

2. Stage one: the speaker encoder

Before any text exists, the system has to answer a narrow question: what makes this voice this voice? A speaker encoder answers it by compressing a few seconds of audio into a single vector — typically 256 to 512 numbers. This is the speaker embedding.

Training teaches the encoder to make two properties true. Different speakers must land far apart in that vector space. The same speaker, recorded on a different day, through a different microphone, saying different words, must land close together. Nothing in the training objective mentions pitch or timbre; the network discovers whatever features achieve that stability.

This is why a 30-second sample is enough, and why an hour is not dramatically better. The encoder needs enough phonetic variety to estimate a stable point, not more data per se. A minute of conversational speech covers more of the phoneme inventory than five minutes of someone reading a single paragraph at a constant pace.

It is also why sample quality matters more than most people expect. Room noise and reverb are not "extra information" the model can ignore — they shift the vector, because the encoder was trained to treat consistent acoustic conditions as part of the speaker.

3. Stage two: the text front-end

Text has to become sound units before it can be spoken. The front-end normalises the input — expanding numbers, currencies and abbreviations into words — then converts the result into phonemes, the inventory of distinct sounds a language uses.

Phonemes are where multilingual support actually lives. Each language has its own inventory, and a model trained across 83 languages is trained to map several inventories onto one shared intermediate space. The speaker embedding is plugged in independently of that space. That is the mechanism behind a claim that sounds like marketing but is architecturally true: clone once in English, then generate in Japanese, with no Japanese recording from that speaker at all.

The accent you hear in the output comes from the sample, not from the text. If the source speaker had an American accent, Japanese output will carry an American-accented rendering of Japanese phonemes. A native-sounding Japanese clone requires Japanese source audio — the architecture cannot invent an accent it never heard.

4. Stage three: the acoustic model

Now the two inputs meet. The acoustic model receives the phoneme sequence and the speaker embedding, and predicts a mel spectrogram — a time-by-frequency map of energy, roughly 80 to 100 frequency bands, updated every few milliseconds. It is not audio yet; it is a description of what the audio should contain.

Alignment is the hard part. The model must decide how long each phoneme lasts, and that depends on speaking rate, which depends on the voice, which depends on the sentence. Early systems needed an explicit aligner trained on transcribed audio. Modern architectures learn alignment implicitly through attention, which is why they can handle a comma that should become a 400 ms pause without being told.

Duration is also where the prosody of your sample leaks through. Two speakers given identical text will produce mel spectrograms with different timing, because the model learned each voice's habitual pace along with its timbre.

5. Stage four: the neural vocoder

The vocoder turns the mel spectrogram into a waveform: 24,000 or 44,100 samples per second of audio, each a number describing air pressure. Doing this well is why neural cloning sounds clean and parametric synthesis did not.

Older vocoders used signal-processing heuristics — phase reconstruction, source-filter models — which is precisely where the buzz came from. A neural vocoder instead generates the waveform with a network trained to produce audio that, when analysed, matches the target spectrogram. The high frequencies that carry breath, sibilance and room tone are generated rather than approximated.

This stage is also the most expensive computationally, which has a direct consequence for pricing, covered below.

6. Where the cost actually comes from

People often assume speech generation is billed by audio length because audio is the output. Architecturally, the cost driver is different, and knowing it makes pricing legible.

The acoustic model's work scales with the number of phonemes — that is, with how much text you feed it. The vocoder's work scales with output duration, but modern vocoders run close to real time or faster, so it is not the bottleneck. Doubling the text roughly doubles the compute, which is why billing per byte of input text tracks the real cost more closely than billing per minute of audio.

At Resona, speech is billed at 4,200 credits per 1,000,000 UTF-8 bytes of input, and cloning a voice is 100 credits charged once per voice. A 1,000-word English script is about 6,000 bytes — roughly 25 credits, or about 50 cents. Chinese and Japanese text costs three bytes per character, so the same visible length costs more, which reflects how many more phonemes those scripts carry.

7. What this means if you are cloning something

Optimise the sample for variety, not duration. Two minutes of the person talking about different things beats five minutes of one rehearsed paragraph.
Fix the room before the microphone. A quiet untreated room with a phone beats a noisy room with a condenser mic, because reverb and background shift the embedding.
Expect accent transfer, not accent creation. Cross-language output inherits the sample's accent. Cast accordingly.
Test with two styles. Generate one neutral and one emphatic line. If both read as the same person, the embedding is stable; if only one does, the sample is too uniform.
Do not expect mannerisms. Filler words and verbal tics are not phonemes in your text. Write them in if you want them.

Ready to hear the pipeline on your own voice? Open the Resona console, record 30 seconds from your browser, and compare the clone against the original. Cloning is 100 credits per voice, billed once and reusable afterwards; speech is billed by the byte as you generate it. There is no subscription, and credits do not expire. If you want the shorter version of this article, read what AI voice cloning is and how it works; for the recording side, see the sample recording guide.

All rates are listed on the pricing page: cloning is 100 credits (about $2) once, speech is billed by the byte, and credits never expire. You can run the whole pipeline in the Resona console from a browser tab.

New to this topic? Read the full Voice Cloning Guide.