How Does Voice AI Work? STT, LLM, TTS and Telephony Explained

Every time you call a business and a voice AI agent picks up, three things happen in less than a second. First, the agent listens, and speech-to-text (STT) software writes down what you said, word for word. Then it thinks, as a large language model (LLM) reads those words, works out what you need and picks a reply. It's the same kind of AI that powers ChatGPT. Finally, it answers, with text-to-speech (TTS) software turning that reply into a natural voice. Then the loop starts over with your next sentence.

STT, LLMs and synthetic voices have each existed for years. The hard part is chaining them fast enough for a live phone call. Each step has to start before the previous one is done, and the agent has to know when you've finished talking, and when to stop if you cut in. Let's look at each piece.

How does voice AI work? The ear, the brain and the mouth

A voice AI agent answers or makes calls on behalf of a company. It understands what the caller needs and takes action, like booking an appointment or updating a customer record.

Think of it as a person on the phone, with an ear to listen (STT), a brain to think and decide (LLM) and a mouth to respond (TTS). A phone line connects the caller to the agent, and a timing layer coordinates when it speaks. This loop repeats dozens of times in a single call.

Notice that only the brain interacts with the company's tools, like its calendar or customer database, while the timing layer monitors the entire loop.

How does your call reach the agent?

When you dial, your call travels over the regular phone network, just like any other call. On the company's side, it reaches the agent in one of two ways, either through an internet-based phone connection (often called SIP), or by being forwarded from the company's existing number or phone system. Your voice then travels in tiny audio packets, each a fraction of a second long, back and forth.

Vatel connects over SIP trunking or call forwarding, so you keep your current numbers and carrier. It works with phone systems like 3CX, Yeastar, Asterisk, Cisco Webex and Microsoft Teams.

Is anyone talking?

Before the agent tries to understand anything, it has to know whether you're actually speaking. A small, fast model listens to the incoming audio many times per second and sorts it into speech, silence or background noise. Only speech moves on to the next step.

This check matters more than you'd think. It's also the agent's first clue that you've stopped talking. Set it too sensitive and the agent jumps in while you're mid-sentence. Set it too relaxed and you're left waiting.

The ear turns your voice into text

Next comes STT. Its job is simple to describe. It writes down what you say. What's impressive is that it does this while you're still talking. It makes a first guess after a few words, then corrects itself as more of your sentence comes in. That way, the rest of the system can start preparing before you've even finished.

This is also where a lot can go wrong. Accents, background noise, unusual names and phone numbers all test the ear. If it mishears you, the brain will answer the wrong question, no matter how smart it is.

The brain understands and decides

The brain is the LLM. It reads what the ear wrote down, along with everything said so far in the call and the instructions the business gave it. From there, it works out what you want, picks out the details that matter (your name, a date, a file number) and decides what to say next.

The brain can also take action. In the middle of your call, it can check a calendar, look you up in the company's customer system or open a support ticket. Vatel agents connect to tools like Salesforce, HubSpot, Freshdesk and Stripe, plus 35+ other apps. To make sure it follows the company's rules, like opening hours or a cancellation policy, a business agent usually works from a defined call flow instead of one long list of instructions.

At Vatel, that call flow is built in a visual editor. Each step of the call, like speaking, branching on what the caller wants, capturing a detail or transferring to a human, is a box on a map, so the agent can't wander off script.

The mouth gives the answer a voice

Once the brain has a reply, TTS turns it into spoken words. Here's the clever part. The mouth doesn't wait for the full answer. As soon as the first words are ready, it starts talking, while the brain is still writing the rest. That overlap is a big reason the reply feels quick.

The voice itself is also a choice. A business can pick the tone, gender, language and accent that suit its brand and its callers.

At Vatel, that includes Quebec French voices, so French-speaking callers hear an accent that sounds like home.

Timing tells the agent when to talk and when to listen

Wrapped around the ear, the brain and the mouth is a coordination layer that decides who speaks when. Think of it as the agent's manners. It handles two tricky situations that make or break a call.

• Knowing when you're done. If you say “Let me find my number…” and pause, you haven't finished. A simple timer would either cut you off or leave an awkward silence. A good agent estimates whether you've really wrapped up your thought.
• Handling interruptions. If you talk over the agent, it should stop almost instantly, listen to you and remember where the conversation was.

Vatel agents do both. They wait until you've really finished, and they stop as soon as you cut in.

One sentence, from start to finish

Let's follow a single sentence through the whole loop. Say you call and tell the agent, “Hi, I need to move my appointment to Thursday.”

1. While you're talking, the ear is already writing “Hi, I need to move my…” A moment later, it has the full sentence.
2. When you stop after “Thursday,” the timing layer decides you've finished, rather than just pausing.
3. The brain understands what you want (to reschedule), notes the key detail (Thursday), checks the calendar and finds two open slots.
4. The mouth starts with “Sure, I can move it to Thursday” while the brain is still preparing the rest, “I have 10 a.m. or 2 p.m. Which works best for you?”
5. If you cut in with “2 p.m. is perfect,” the agent stops talking, listens, books the slot and confirms.

From your side of the phone, it just feels like a smooth chat with a receptionist.

How fast does voice AI need to be?

Fast. A voice AI agent should start answering in well under a second. Think about how you talk with people. You usually reply within a few hundred milliseconds. A study of conversations in 10 languages found that the average pause between turns varies by only about 250 milliseconds from one culture to another (Stivers et al., PNAS, 2009). On the phone, anything longer starts to feel like hesitation, and callers repeat themselves or talk over the agent.

So where does the time go? Surprisingly, not mostly to the ear or the mouth, which are fast. Most of the delay comes from two moments, deciding that you've really finished talking, and waiting for the brain to come up with its first words. That's where a well-built agent saves the most time. It also helps that the steps overlap instead of waiting for each other.

Step	Where time goes	Share of the delay	How it is kept short
End of turn	Deciding the caller has finished	Large	A model that predicts the end of a sentence, not a fixed timer
The brain	Producing the first words of the reply	Large	Fast models and short, structured instructions
Phone line	Audio travelling to the agent and back	Small	Servers close to the callers
The ear	Transcribing the last words	Small	Streaming: most of the sentence is already transcribed
The mouth	Producing the first second of audio	Small	Starting to speak before the full reply is written

Vatel agents typically answer in about 600 milliseconds.

When the brain has to look something up in a slow system, a good agent fills the gap the way a person would (“Let me check your file”) instead of leaving you in silence.

What's the difference between a voice AI agent, a voice assistant and an IVR?

These three get mixed up a lot, so let's clear it up. A voice AI agent holds a real conversation on a company's phone line and gets business tasks done. A voice assistant, like Siri or Alexa, helps one person on their own phone or speaker. An IVR is the classic phone menu you've probably been stuck in (“Press 1 for sales”). It sends calls in the right direction but can't hold a conversation.

	Voice AI agent	Voice assistant (Siri, Alexa)	IVR phone menu
Who it serves	A company's callers	One person, on their device	A company's callers
How callers interact	Natural conversation	Short spoken commands	Keypad or set phrases
Can it complete a task	Yes, in the company's systems	Personal tasks (timers, music)	Routes the call, rarely more
Handles off-script questions	Yes	Partly	No
Available 24/7	Yes	Yes	Yes

Why each part should be swappable

Here's something most people don't realize. The ear, the brain and the mouth are usually three separate technologies, often made by three different companies. Each one keeps improving every few months, and not at the same speed. The best ear for Quebec French isn't always the best one for English, and a faster brain might come out next quarter.

If your platform locks all three together, you're stuck with whatever it picked. With a modular setup, you can upgrade one part without rebuilding the whole agent, whether that's a better French voice, a cheaper model or more accurate transcription. Swapping a part is the easy half, though. The hard half is making sure timing, interruptions and transfers still work afterward. Vatel connects to several providers at each layer, such as Deepgram and Speechmatics for STT, Google and Anthropic for the LLM, and ElevenLabs or Cartesia for the voice. Vatel's team takes care of both halves for you.

Want to dig deeper? Compare open stack and all-in-one platforms, then see how to choose STT, TTS and LLMs.

Where does call data go at each step?

Every step of the loop handles information about the caller, from the audio on the phone line, the transcript in the ear, the conversation in the brain to the recording once the call ends. If each step is run by a different provider, that information can be processed in several places, sometimes in several countries.

So for a Canadian business, it's worth asking two questions, not one. Where is the data stored, and where is it processed during the call? Vatel fully complies with Quebec's Law 25, and can set up your agent so that hosting, STT, the LLM and the voice all run in Canada. Its SOC 2 Type II certification is in progress.

Can voice AI speak more than one language?

In Canada, plenty of callers start in French and switch to English, or the other way around. For that to go smoothly, the ear has to understand both languages and local accents, and the mouth has to reply in the right language, with a voice that sounds natural.

Vatel agents are multilingual. They detect the caller's language, switch mid-call when the caller does, and understand the Quebec accent.

Common misconceptions about voice AI

A lot of confusion comes from judging voice AI like a chatbot or a recording. On a phone call, speed and timing matter more than anything else. Here are four myths worth clearing up.

• “It's just three tools plugged together.” The ear, brain and mouth can be bought off the shelf. Getting them to work together in real time, with good timing, smooth interruptions and clean transfers, is the real work.
• “A bigger AI model makes a better agent.” On a call, how quickly the first words come matters more than raw brainpower. A fast model with clear instructions usually beats a slower, larger one.
• “The voice is the hardest part.” Today's synthetic voices sound very natural. What gives an agent away is timing, like long silences or talking over you.
• “We'd need to train our own AI model.” Rarely. An agent is tailored through its call flow, your company's information and its connections to your systems, not by training a new model from scratch.

What to ask before choosing a voice AI provider

A polished demo doesn't tell you much. These eight questions will show you how an agent behaves with real callers.

1. Can I hear a real call, not a scripted demo?
2. What is the median response time on a real phone line?
3. Which STT, LLM and voice do you use, and can they be changed?
4. Where are audio, transcripts and recordings processed and stored?
5. What happens when a caller interrupts the agent?
6. How does the agent hand a call to a human with the full context?
7. Which of our systems can it connect to, like our phone system, CRM or calendar?
8. Who builds and maintains the call flows as our needs change?

Want to hear the ear, the brain and the mouth handle your own calls? Book a demo

FAQ

What is voice AI?

Voice AI is technology that lets software hold a spoken conversation in real time. It listens with speech-to-text, thinks with a language model and answers with text-to-speech. On a business phone line, it can answer calls, understand what callers need and get tasks done.

Is voice AI the same as a chatbot?

Not quite. A chatbot works in writing, on a website or in an app. A voice AI agent works on the phone. It adds an ear to understand speech and a mouth to answer out loud, with the same kind of brain in between.

Is text-to-speech generative AI?

Yes, modern text-to-speech is generative AI. Instead of stitching together prerecorded sounds, it creates new speech from text. That's why today's AI voices sound natural, with real rhythm and intonation.

Can voice AI understand accents and French?

Yes, as long as the ear is chosen with the callers' languages in mind. Some STT models handle Quebec French far better than others, which is one more reason to keep each part swappable.

Do callers know they're talking to an AI?

They should. Saying so at the start of the call builds trust, and privacy laws such as Quebec's Law 25 set rules on transparency and on recording calls.

Vatel agents tell callers they're speaking with an AI at the start of the call.

Can a voice AI agent transfer a call to a human?

Yes. A well-built agent knows when to hand off, transfers the call and passes along what you've already said, so you never have to repeat yourself.

At Vatel, transferring to a human is a step built into the call flow.

What happens when the agent doesn't know the answer?

It should say so and offer a next step, like transferring you or arranging a callback, instead of guessing. Clear rules in the call flow keep the brain from making things up.