Designing Voice Agents for Real Conversations - Chintan Agrawal & Daniel Wirjo, AWS transcript

Hey everyone. I'm Chintan and I have my colleague Daniel with me. We are solutions architect on the AWS APJ startup team. So today we are going to talk about something we've been working on for a while. Which is turn taking in voice agents. What I mean by that is how does the system actually know that you have finished speaking and it's time for the agent to respond? These are all audio engineering problems. They are not LLM problems because you can have the perfect model, perfect track but the experience still might feel broken if the turn taking is off.

So we are going to walk you through the concepts and three different approaches to solving this. And Daniel is going to you know make it more concrete by running all the three demo for you. So to kick things off uh we'll start with the 200 millisecond constraint which is basically the physics of why voice is hard. Then we look into the pipeline architecture, what are the components, where turn taking actually lives. We'll then look at three level of solving this problem.

Going from simple silence detection all the way up to running your own turn detection model. And then we'll also talk about the latency, the budget and production issues that we've seen. So but before any of that, I just want to show you the problem statement more clearly because you have the same user. You have the same sentence on the both side but the user outcome is very different because on the left when the user says I want to fly and midway they try to correct themselves, the agent just does not notice.

Because it keeps going on for almost 2 seconds while the person is sitting there trying to get a word in. On the right side, the same exact interaction is happening, but the agent is able to capture the interruption in under 200 milliseconds. And immediately backs off so the user can now speak. Both the scenarios, the LLM was identical. It was the same model. It was the same prompt, but the difference in the user experience is so different.

And the difference is purely because of the audio pipeline. And how fast does it notices like someone else is talking and it knows when to shut up. Like that's the problem we're trying to solve today. So, we'll cover this 200 millisecond constraint because 200 milliseconds is how fast humans switch turns with each other in a conversation. And the implications are pretty brutal because at 800 milliseconds, things start to feel off.

While at 1.5 seconds, your user just hang up on you because you have like your HR chat agents might get 5 seconds to respond and nobody will care. But with the voice agents, you don't get that luxury. We've seen recently like Salesforce was able to pivot the pipeline and they published their results in March 26th. But even there, best measured response time was 755 milliseconds. So, that's like almost 4x slower than how humans naturally take turns when talking.

So, the question here becomes how do we close this gap and where we cannot close it like at least how do we get the turn taking right? So, the user experience does not feel as bad as the raw numbers might suggest here. So, if you're working with a budget of 755 milliseconds and every millisecond count, you also need to understand what's actually in the pipeline. Like where does the time go and where does this turn taking fit?

So, the pipeline itself you probably might know the main components, the STT, the LLM, the TTS. You have the audio in, text out, text in, audio out. But the piece that's often missing like from people while people are thinking about this is this voice activity detection. Commonly called as VAD. It's a tiny component that sits right at the front and it is the one that controls turn taking. Its job is to basically detect whether the user has stopped talking or not.

And the two other thing that you set alongside main pipeline are also very critical for turn taking. So, you have the smart turn detection which watches the VAD signal. Plus the audio features and then it has to make that decision that whether should it respond now or should it wait because the user might still be talking. And the interruption handler is triggered when someone barges in. So, what it'll do is it'll propagate a flush of downstream like the TTS would stop and the LLM generation need to be canceled so that the pipeline is ready for the new input within about, you know, 50 ms.

So, if you remember I was talking about like how there are different levels to, you know, building this turn taking in voice agents. So, we'll start with a level one. Level one is your Silero VAD. So, this is a simple component which you fully own and like a lot of production system even today we see are running agents just by using this. So, coming to Silero VAD, it's a small 300,000 parameter model. It takes in like a short term Fourier transform.

It has a and it takes raw audio and it'll convert it into a spectral features. It has four convolutional layers to pick up the pattern and an LSTM that gives it memory so across the frame so it's just not looking at one chunk in isolation. And then it has a sigmoid that gives a probability of speech and it's like a very small 2 megabyte model. And the one parameter that minimum silence millisecond is basically the entire user experience of level one.

If you keep it very low, the agent is going to be snapping like it will cut people off while they're still, you know, thinking. But if you keep it very high, the agent might become super patient and it will never interrupt but you might get to uh get like a dead air where people will wonder if the agent is still connected or not. So there's like no universally right answer. It depends on the domain. If you're working on it from a perspective of a sales agent, you probably want it to be like 200 milliseconds.

If you're in a domain where you need some to give some time to your user to respond, it could be like 1 uh thousand to 1,200 milliseconds. It depends entirely on what you're building. So VAD works well like that's why a lot of people are using it and it's generally good at what it does that is to return fast response to the basic question of whether there's someone talking right now. But uh there are things like it was never designed to handle because there'll be situations where just knowing that is there a silence will not be enough because you also need to answer these questions of how long to wait like uh the silence duration.

If someone is pausing for 300 milliseconds or 400 milliseconds, VAD is seeing the same exact thing. It has no idea like whether the person is catching up their breath or whether they've completely finished their thought. So like they cannot distinguish that. And the second scenario is like, what if they, you know, the user talks over the agent, like, it barges in. It does not know how to handle it. And the same silence can have a different intent.

And it can be like completely different things. It could be like a completed sentence, it would be an incomplete thought, it could be like a thinking pause. Or maybe it could be something as basic as like a backchannel acknowledgement. Uh the situation of the silence is same, but the intent again was very different in all the scenarios, but VAD considers all four equally, and it literally cannot tell them apart. So, those are the limitation of level one.

And with the level two, basically what you get is that you let the STT service tell you like when the turn is over. So, Cartesiant does turn detection right inside their STT websocket. So, when they stream your audio in, the server would handle both the transcription and the turn detection together, and it will emit event when it thinks like the turn is done. Similarly, Deepgram does the same thing with their what they call as their I think at the endpointing.

And their P50 latency for Cartesiant is about 300, and it's about 250 for Deepgram Nova 3. Both of these are also working really well. Like, we see a lot of customers use that. Because they are using like full audio signals plus some linguistic context to make these decisions. So, which is like much more information than your VAD will ever have. However, the trade-off is with the transparency. Because when it is working, it works great.

But when it misfires, or when it cuts someone off at the wrong moment, then you like have no way to figure out because there's no law that will say uh it was done because of this reason, because of the what was what it saw. And that decision was made inside someone else's server, and you just got to live with that, basically. So, that brings us to level three. So, in level three, you keeps a little VAD running locally for basic signal that is there a speech.

And you add Smart Turn on top of it, which is a small model that runs during silence. So, Smart Turn V3.2 is the latest model, like, and uh let me share some numbers with you. And it has about 58.9% recall and 68.4 percent precision. So, what this means is that about six out of 10 time when someone finishes a sentence, Smart Turn will be able to catch it quickly. But the other four times, it might not be confident enough, but that's also fine because you still have your VAD timer, which is running underneath as a safety net.

If the Smart Turn doesn't fire, your stop second will kick in, like, at the configured latency anyway. So, it could be like 300 ms. So, in that scenario, you are never stuck waiting. You just, like, get the fast respond when the model is confident. And you get a slightly slower but a safer respond when it's not. The other number on this list is from, like, a Meta's published paper earlier this year, um around March. There, they had reported a higher recall at 87.7%, but they haven't released the code, so you really cannot deploy it.

Smart Turn is BSD-2 licensed, and it's a 8-MB small model. You can pip install it today as well. So, yeah, like, that works out better for a lot of our customers. We decide when to start talking. But like, what about the other direction? What happens when the user interrupts and while you are already, you know, the agent is talking? So, on the left side, you can see what happens mechanically. Like, user opens you mouth, the windy will pick it up in 32 milliseconds and then then like 15 milliseconds, the entire pipeline is flushed.

TTS stops, LLM cancels, everything is clean. Like, the user never hears the agent. You know, that part is taken care by Pipe Cat. But, the important thing is like, should it have stopped there on because think about a normal conversation when someone says, "Yeah." Like, if they send an acknowledgement while you're speaking, you don't stop, right? You know, they're just agreeing, but if you they say like, "Okay, wait, no.

Like, that was wrong." You have to stop immediately. Same thing here. Like, the red is the it's a correction. Stop on everything. They amber could be just some sort of a filler background noise. But, you want to go ahead and keep going, maybe in a bit quieter. Blue means like a signal for cough for a background noise. So, you can just ignore it and finish your sentence. So, at level three because you own this classification, you can start distinguishing between a real interruption and someone just, you know, going um.

So, today most of these systems stop every time, but this is a piece that's improving. Like, we are seeing a lot of incremental improvement in this. So, by now that you've seen all three levels and if you look at these three Python files, pipeline is almost identical in all of them. The only thing that differs is how you answer all of these questions. When is the user done talking? In file one, you pass in Celero VAD analyzer and that says the user had been silent for 300 milliseconds, so you can assume they're done.

In file two, you swap it with the Cadence turns STT service class and now the service is going to tell you when the turn is over. And in file three, you keep your Celero VAD analyzer, but you also add like a local smart turn analyzer version three, which is sort of a small model that runs during your silence and it's going to look at the prosody and it's going to look at prosody and intonation to decide whether that pause meant I'm done or I'm still thinking.

So, you have the code-wise the same pipeline, but the configuration changes, which you know, leads to completely different behavior. So, to summarize your three levels, where our level one was Celero VAD, so you basically own the silence detection completely. Level two, you let the STT provider handle it for you. And in most cases, it it will speak smarter, but then you cannot see what happened inside it. Level three is your VAD plus smart turn, so where we basically own everything, like you have full portability.

And now coming to the latency piece, because we've been throwing around these numbers, 755 milliseconds, 1.3 seconds, but let me show you where all the time actually goes in. So, this is a breakdown from Quintela, who was a co-founder of Daily.co and is the person who created Pipe Band. And these are some of the numbers based on their production measurement. So, you have the numbers for making an which is about 40 ms.

Then you have network and jitter buffer of 52. So, these are physics. Like you can't really change them much. Then you have your transcription plus end pointing. That's about 300 ms. Again, that's your STT doing its thing. LLM time to first byte, and this is generally the dominant bottleneck because in a typical API setup, you're looking at 500 to 650 ms depending on what model you're calling and from which cloud provider, which region.

And then you have your network out and playback TTS. That's it's going to take about 120 to 90 85 ms. So, you're totally looking at roughly 1,100 to 1,300 ms in a standard setup calling cloud APIs. Now, the Quindos team had demonstrated about 500 ms total voice-to-voice by co-locating all models in the same GPU cluster. Because they were able to eliminate the network hops within the service. And it's sort of an achievable floor because if you're willing to invest in infrastructure, but for most of the developers today calling APIs, you're somewhere in the range of 800 to 1,300 ms.

And the key insight to note here is that the STT, LLM, together eat about 2/3 of the is latency budget. So, these are the really the only two levers that we have to move this latency metric needle in a meaningful manner. So, coming to like which LLMs are actually fast enough for this? We benchmark current models specifically for voice, and the target we kept it as under 700 ms time to first token. Because anything slower will push our total response past where the user is going to start noticing.

So, for now, like recording in June 2026, what we've observed Nemotron-3 Ultra gives us a 529 P50 latency. GPT-4.1 at about 536. But also, like what matters more in voice uh than P95. So, which LLMs are actually fast enough for this? We benchmark current models, specially uh for voice, and So, coming to the So, coming to the LLMs models, which are actually fast enough for the We benchmark current models well for voice.

Uh so, these benchmarks are from June 26, and the target that we kept was under 700 ms for time to first open, because anything slower would push our total response time past the limit, like where users is going to start noticing. So, for now, the Nemotron-3 Ultra at 529 P50, we observed GPT-4.1 at 536, again P50. But what matters more in voice than maybe anywhere else is that the P95 tail. Because GPT-4.1 was great at P50, but it spikes to 1.7 at P95.

And it was even worse for Claude 3 that it hit over 4 seconds and because in a conversation you cannot average this out because one slow response and your entire flow is gone. And there's one more dimension that people often miss, which is the multi-turn trip because after 15 or 20 turn, sometime model starts ignoring parts of the system prompt. They might get to verbose, they go off spirit. Because in that's fatal, you can't just dump a wall of text on someone if the instruction following degrades across turn.

Then you have to do some sort of a context pruning or session resets. So a few production issues that we've seen that false interruptions significantly increase your escalation rate. Because when agents cut people off incorrectly, then users are more likely to request for a human in the loop uh support. And a 755 ms that's still I think the best measured voice-to-voice for a cascaded pipeline like the ones which we propose.

We're nowhere near the human speed of interaction, which means the return taking also needs to do a lot of heavy lifting to make the experience feel better than the raw numbers it would suggest. And at 58.9% recall on smart turn, that's actually the best deployable turn detection you can get today and because although it still leaves like four out of 10 turns to be filled by a uh VAD timer, but this space is going to improve a lot over the next year few years and uh we'll have better model.

And on infrastructure running all of this in production is also hard because five systems that all scale differently and can also fail differently. You can explore Picovoice Cloud for managed hosting where your focus is on agent logic or the AWS guidance reference architecture for enterprise deployments with all the enterprise guardrails. So yeah, like uh Uh that's it from me. Everything uh we've discussed or mentioned is linked on the screen that I pulled benchmark.

Okay, over to you, Dan. >> Thanks, Jinson. Hi, everyone. Uh my name is Daniel Weijia. I'm also a solutions architect with the AWS startups team. So I think um Jinson walked you through a great overview of some of the key challenges when building voice agents. So what I'm going to do in this session is just to walk you through a hands-on demo um just to make the concepts a bit more concrete. Um hopefully um you can kind of um incorporate some of the learnings into your own voice agent pipelines as well.

So what we'll be going through is just this Picovoice Cat 10 detection demo, which is just a simple repository that we've created. Um we'll talk through three examples. So one with um silence detection, another with um built-in 10 detection in the speech-to-text model. And then finally, um we'll go through the open smart 10 model as well. So this is really useful when you either want full control of your um voice pipeline just for from a compliance perspective or if you want to also fine-tune and customize your own 10 detection model based your own based on your own data.

So yeah, let's get started. So let's just go through the source code of the first example. As you can see here, um using Silero is pretty easy in Pipe Cat. All you have to do is import the relevant libraries, and you can configure the parameters as well. Um I haven't actually uh configured any of the parameters, so these are just the default, but you can change the parameters like the confidence to trigger, the silence duration before the turn ends, and the minimum audio volume as well.

And in terms of the voice voice pipeline, it's pretty standard. So, we've used um an STT model, the LLM, and also a TTS model here. We've just defined it as the Cartesia model for STT. Same with the Sonic model for TTS, and as you can see, we've used the voice that sounds like a a British lady. And for the LLM in the voice agent's brain, we've just used the Anthropic Claude like cool model. So, we'll see um what that looks like, and I'll spin up this example right now.

So, So, if you haven't used Pipe Cat before, one of the nice things is that it can spin up a local prototyping environment. Really useful for playing around with your voice agent pipeline. So, let me just see if this now works. Okay. So, as you can see, it will come up with a um local uh development environment. So, what I might do is just connect to the voice agent, but before I do that, I might just mute just in case um I'm interrupting the voice agent.

It should come up with like an opening greeting, and then I'll test it with some uh interactions to see how the turn detection is working. >> Hey there, I'm your travel assistant and I'm excited to help you plan an amazing trip. So, where are you thinking of going? Anywhere in the world you've got your eye on? >> As you can see, the assistant's gone ahead to do the opening greeting and on the right here in the terminal, I've just got um some real-time logs just so we can see what's happening.

So, what I might do now is just I'm going to pretend that I'm uh thinking about my travel destination and see how the voice agent behaves. So, I'm thinking of going to um >> I'm all ears. Where are you thinking of heading? >> As you can see, I haven't really finished my uh sentence yet, but the voice agent is already responding. So, you can see how this works in the debug logs and as you can see, there's an end of turn complete here and the reason for that is due to the stop seconds parameter, which is um set to um 300 milliseconds, it looks like here.

So, that's why it's gone ahead and ended the turn. Um what I might do is I might just exit this local server and try the second example. So, let's just clear that and I'll refresh that. Okay, so I'm going to try the second example here. And in this example, we'll use a um speech-to-text model which has turn detection. So, we'll see how the voice agent behaves. Okay. I'm going to try uh the same example, so let me just mute myself here. >> Hey there.

I'm your travel assistant and I'm excited to help you plan an amazing trip. So, where are you thinking of heading? Are you dreaming of a beach getaway, a city adventure, mountains, or maybe somewhere completely different? >> All right, let's see how the model behaves this time. Hi, I'm thinking of going to um As you can see, the model is pretty smart. It now can detect the turn, so let's see if we can find out what's going on in the debug logs.

So, um let's see if it emits uh some of those turn taking events here. So, as you can see, um the Katisha Inc. model does emit some turn events here. And as you can see, it hasn't really completed the turn. It hasn't sort of um responded back with any response. But, maybe I can unmute myself and then resume and get to the destination. So, Sydney? >> Oh, Sydney is fantastic. You're going to have such a great time there.

Are you thinking about when you'd like to go? And how long are you planning to stay? That'll help me figure out the best flights and hotels for you. >> So, as you can see, once the model detects that it's a full complete sentence, is able to then respond. Now, in practice, you would also incorporate um some silence detection there because as you can see, I was quite silent. Um I was muted for a while and you'd probably want the voice agent to respond in practice.

Um having like voice uh activity detection or silence detection is also useful for um interrupting the voice models um um or what you call barge in as well. So, definitely this is just a a quick demo, but in practice, you would probably combine both the turn detection model and the silence detection. Um just for the final demo, I might quickly show the smart turn model as well and see what that looks like. Um it will behave very similar to to this.

It's just that um the model will be running on your uh either your local machine or um on a GPU uh environment as well if you want to deploy that into your into the cloud. So, let me just maybe clear this just to avoid confusion. And then I'll use um I'll refresh this as well. And I'll start now the smart turn example. I'll refresh this to see if we can Okay, there you go. It's starting it. And I'm just going to do the same example.

So, I'll mute myself again. >> Hey there. I'm your travel assistant and I'm excited to help you plan an amazing trip. So, where are you thinking about going? Are you dreaming of a beach getaway, exploring a new city, or maybe something else entirely? >> I'm thinking of going to um >> I'm all ears. Where are you thinking of going? >> Now, let's take a look at the um both the silence detection, which is incorporated into this example, and also the turn detection model.

Hopefully, there are some um debug logs that we can see. As you can see, um it's not detecting that I've completed the turn. The end of turn state is incomplete, but it did complete it um due to the silence that I've um I've um incurred there. Um so, that there was some silence, and therefore it sort of kept going with the with the um next turn. What I might do is see if I can do a complete sentence and see if, you know, um it can be detected through the local smart turn model.

I'm actually thinking of um going to Sydney. >> Oh, Sydney's fantastic. I'd love to help you plan that trip. So, tell me, what time of year are you thinking about going, and how long are you planning to stay? Also, are you traveling solo, with a partner, or with family? >> Okay, fantastic. So, the voice agent was able to respond as soon as I uh completed the sentence, but let's see if that was actually intentional, or was that um something that um you know, can be detected through the smart turn model.

So, let's have a look here to see if we can find the smart turn um debug logs. As you can see, this line here, I'm sorry about the legibility of the highlight, but effectively this here says that um the end of turn state in is complete, and the smart turn model is actually classifying that there is a high probability that that um the sentence is complete and therefore it's okay to go to the next turn. So that's in a nutshell how turn detection works.

Hope that was helpful.