Deep Learning with PolyAI
PolyAI's CEO/co-founder Nikola Mrkšić and team invite guests to candidly discuss trends and tech in AI, voice throughout the enterprise, and nailing the customer experience.
Deep Learning with PolyAI
Can AI really hear a call the way a person does?
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
PolyAI just launched Dialog-RSN-1, its first audio-native model, and your host Nikola Mrkšić sat down with its builder, Matt Henderson, to unpack why it’s a game-changer for building voice agents.
Most voice AI either flattens a call into a transcript and loses the audio, or goes fully speech-to-speech and gives up control of the voice. Dialog-RSN-1 does neither. It hears the raw audio directly, decides when to speak, and keeps text-to-speech separate so the voice stays under your control, all in under 300 milliseconds.
Nikola and Matt get into what audio-native really means, how auto-reasoning keeps it fast, and why it beats every other real-time model on quality and quickness. Hear the full episode, and see how PolyAI builds dialog agents that hear the whole call at https://poly.ai?utm_source=youtube&utm_medium=podcast&utm_campaign=podcast&utm_content=podcast
Talking about reasoning, how does it do auto-reasoning? So dialogue reason, there's a contention in the name. So dialogue means it needs to be real time, otherwise you can't have a dialogue. And any like big LM you're using day to day is using reasoning, but it's slow. So how do you do both? The big thing we're doing is auto-reasoning, where the model will only reason when it would help. And so in practical terms, like 20% of requests get reasoning. And then when it does reason, we train that to be efficient and short.
SPEAKER_01You couldn't really say that it's the best dialogue model that can reason in under 300 milliseconds.
SPEAKER_00Thanks, Nicholas. Thanks for having me back on. Today we're going to talk about our new model dialogue reason one. I think it should be quite a beefy topic for the podcast because it's our first audio native model. So instead of having a cascaded system where you got speech recognition bolted on the front, LLM is then fed with just one best hypothesis, typically, from what the speech recognizer thought the user said. Because oftentimes LLM just wishes it could hear what the audio was. Was that a mispronunciation? Like, are they talking to the same person? Is that background noise? Am I listening to music? Whatever. And that dialogue reason one can, because it is an audio LLM. It's not only listening to the audio in order to decide how to respond, but it's also deciding when to respond. So we're getting that natural fluid turn taking that you get from speech to speech models. But we'll get into a bit later, I guess. It's not speech to speech full. We don't have a lot of the compromises that you have to sort of take if you're going full speech to speech. But I think we get all of the mostly all of the benefits, right? So like true audio understanding, contextual speech understanding, super fast latency and like fluid, comfortable tone-taking, even.
SPEAKER_01Look, I think, you know, just as we talk about dialogue, and 12 years ago, I think was the first time I read a dialogue research paper by a man called Matt Henderson. And it was a dialogue state tracking challenge, too. Also a fun fact for people looking at the model name, as we looked at whether it should be the American or the British spelling, I think that uh your papers, followed by mine, were the ones using the American spelling most consistently. But I think it's, you know, back then dialogue state tracking was about both like memory through turns and about the multiple speech hypotheses that you would then look at to kind of figure out what the model really said. I think it's like so in the spirit of deep learning that it's now just like been gobbled up into one mega model. And mega models are like the theme of every modality. Why not fully speech to speech?
SPEAKER_00Why is it just an audio LLM? For it to say, like yeah, I'm kind of excited that we are doing spoken language understanding again. And we're not just pretending that speech has been solved and feeding into text-based LLM. And a lot of our work back then was doing stuff like that, like consuming the confusion network, for example, and that kind of thing. Why not speech to speech? Or like, well, it is speech in and still text out. So dialogue reason one is outputting either function calls, tool calls, so we go into the agentic loop, or it's like response in text to the user, which goes into text-to-speech. So we're keeping text to speech separate for a few reasons. I guess the first main one is controllability. So if you're adopting a speech-to-speech model, you will be adopting their built-in voices. So you get a handful of voices that you can pick to represent your brand. If it mispronounces the name of your company or like one of the core things you sell, like tough luck, I don't know if you can uh reliably prompt them to pronounce things correctly. I don't think so. So yeah, we get controllability, you don't have to compromise there. I also feel like a lot of the motivation. I like end-to-end models. I think that's where we're gonna go. That's where the industry's going. But the fact is that it's a pain to train those. Like the data you need is just speech and speech. That's annoying. They're also just lagging. They've come a long way, like this calendar year. They're still a bit behind on instruction following calling tools reliably.
SPEAKER_01I think rather than just saying like it's not there yet, there's also maybe less to gain from tying that into the loop. We just did like a podcast episode, the previous one on word air rates, where Oliver Scholson was kind of like showing me things that almost like anti-correlated performance of the end-to-end system with we're there rate. It's not always anti-correlated, but the fact that it could even be anti-correlated shows you just like how much A, it's not solved, B, it's not necessarily indicative. But like at least you know, like having that audio, it can reason in and goes like, hey, my name is Nikola Mrkcich, Nikola is recognized. The blob after is weird, but then it goes and thinks and it's like, hey, something with an each is probably coming because Nikola is the most common Serbian name. And like it actually works reliably. I mean, the other reason one. So I'm astounded that it works. But like with text to speech, what would be like the main benefit of tying it in together?
SPEAKER_00I was thinking is like oftentimes it's kind of like cherry on the cake. It's not this like fundamental problem that you get. If you add audio on the input side, then you fix a lot of problems. You remove the information bottleneck going through text. It can not get tripped up by mispronunciations or like weird new terms or rare names. On the output side, to me, it always feels like, well, okay, it could maybe match your tone or it could match your pronunciation. It's all like good stuff that would drive more human-like natural conversations, but it's no killer use case, I don't think. So I like the direction, making everything end-to-end learnable and having gradients go from what the assistant said all the way back to what you said as like the ideal world. I guess there's a distinction within speech to speech that's worth mentioning as duplex versus more like turn-taking stuff.
SPEAKER_01With dialogue reason one, we have tried to get the benefits of like the duplex world with the sort of like so maybe just for the audience, kind of like can you maybe just like distinguish between like the turn-taking and the duplex approaches?
SPEAKER_00So duplex is basically the system can talk while it's listening, more or less. In practical terms, it means that you use the transformer to like have a constantly streaming channel, both for the user's like input audio tokens and the audio tokens it's deciding to generate. So if you have a caller, you must assign that a GPU and keep it live. So you start just need this, it's a nice solution, but it's an expensive solution. But what it allows you to do is to model anything. That is a probabilistic model, the system can back channel while you're talking. So yeah, at the right times, hopefully. You can interrupt it whenever it can interrupt you whenever. On the other hand, with like turntaking, you maybe have a more explicit state machine. Okay, the user's talking, and then maybe I will take a decision if I should interrupt a sort of fixed time step and that kind of thing.
SPEAKER_01Interrupt is more of a hardware interrupt, like stop, stop, stop, stop, stop, right?
SPEAKER_00Yeah. A duplex model doesn't even necessarily have a concept in its mind about interruption. It's just it would just be generating silence. And then it would generate something that's not silent. So I think that's where it's going. If these models that are out there that are duplex, they're either like very toy, only capable of sort of chit-chat. So there's a VNV, I forget the name. The Persona Plex one. They came up with the one that can do function calling stuff, but it's very limited in its instruction following and ability to like do real, but you know, that's gonna catch up. So with dialogue is on how does it work, how does it do turn taking? We train it so the first token that outputs for any generation after a user audio is its decision of whether or not it should speak. So in under 100 milliseconds, because while they're speaking, we like pre-fill its attention with everything up to what they've said. And then as they speak, we can chunk more in, and then we say, okay, we'll use a simple kind of like energy-based acoustic VAD to say, here's a pause, the user's paused. Do you want to speak or not? And then dialogue reason one will say, Yes, this is a good time for me to speak, or no, the user's still speaking, something like that. And then immediately after that first token, it just generates the rest of this response. So that's like effectively zero time to first token after that.
SPEAKER_01So it's almost like this VAD is like some proto-acoustic model of the old speech recognizer, and everything else becomes one model.
SPEAKER_00That's right. We use this acoustic VAD as a helper, like a dumb small model. Well, so we pick the best one we can, but it's dumb in that it's it's not an LLM, right? It's not billions of parameters. It's not, it doesn't have the full context for the conversation. It's not like dynamically updating based on the context of what it's heard. All this is stuff that dialogue reason one does. But it's fast and cheap to run, and it proposes like reasonable points in time for dialogue reason one to take turn-taking decisions. So then dialogue, the big model, acts as like the brain on top that's acting on a less frequent time scale and then deciding how to like when and and how to reply. That helper models tuned to be super fast. So once you have finished speaking, it can respond super quickly. But you don't have that same compromise we would used to have with cascaded systems where if you pause for long, assistant would be too eager to interrupt you. Because it can hear you, it will just reject that pause saying now they're still talking. What else did we do to get the latency down? So there's a yeah, a whole bunch. And like I guess that's part of what we do is we're optimizing for that user-perceived latency. We're not optimizing for the cost of us to serve it. We just really want to make the latency super fast. So like we don't care really about throughput or you know, multiple. We want to make it any call just sound super fast. Because if you don't have super fast latency, it won't sound natural, right? So we do a bunch of things. So like quantization is a big one. The model is quantization of where training will make that no compromise in quality. We also do some of the tricks I alluded to where like while the user is speaking, we can pre-compute a bunch of stuff. So all of that like context is pre-filled. We do speculative decoding. Have you seen the speculative decoding? I have not. This is like a cool thing where you basically have a small helper model that's like a head on top of some of the hidden layers of the main model. It is asked to predict, say, like seven tokens into the future. What do you think? What does the small model think the big model is gonna say? So normally, obviously, you generate one token at a time, auto-regressive, and then you kind of hit with this loop where you have to wait for that token to be generated to feed it in. If the small model could play guess seven tokens in ahead, then you would just batch those three and you'd make use of the efficient batching on the GPU and you speed things up. How do you know it predicts the right ones?
SPEAKER_01Well, we just train it. So it's almost like the distilled model of the bigger one for predicting the next few rather than the next very many, right?
SPEAKER_00Yeah, exactly. It's distillations. We match like the KL between the big model and the small model. So we train this model to match the you know, the greedy decoding predictions going ahead seven tokens into the future. And on average in production calls, it's correct. Up to on average, four tokens are accepted. So while the big model is generating is actually like accepting, say, you know, on average, four tokens into the future. Part of that's just we're exploiting our specialization. We don't need to train it to like predict really well ancient French poetry or something. It's like our function calling format, our turn-taking format. Dialogue reason when it like cites knowledge citations at the end, and all that these are like formatting tokens it can get, but also just the reasoning itself and the and the output is predictable.
SPEAKER_01You're talking about reasoning.
SPEAKER_00How does it do auto-reasoning? So dialogue reason, like there's a contention in the name, right? So dialogue means it needs to be real time, otherwise you can't have a dialogue. And then reason, like okay, reasoning models, there's a massive breakthrough in LLMs, like do reasoning, just think for thousands of tokens. And any like big LLM you're using day-to-day is using reasoning, but it's slow. So, how do you do both? And uh like the big thing we're doing is auto-reasoning, where the model will only reason when it would help. So, as part of our RL fine-tuning that we do, we generate multiple candidates from the model, and we do some with reasoning, some without. And then we just see if you didn't do reasoning, do you do worse, basically?
SPEAKER_01Uh-huh. So in training, you kind of look around some with without, and then like a wrong thing, it decides on its own whether it should be reasoning or not?
SPEAKER_00Yeah, using those rollouts during training, that we would then train that decision token, should I do reasoning or not, based on like the payoff. And in practical terms, like 20% of requests get reasoning. And then when it does reason, we train that to be efficient and short. Like creative decoding helps because say it generates 50 tokens, that might in reality be like 12 chunks of tokens that get accepted. But yeah, we have like a lost term inside our reinforcement functioning for reducing the length. Basically, we optimize in reinforcement functioning for the added latency. So a P50, a median added latency from doing reasoning is zero. Most cases it reason. So then like P80 is where it like has 300 milliseconds or something. So at some point in the call when it gets to that difficult point, and it wouldn't it do this twice inside one agentic loop, but just do it once, it would add like 300 milliseconds, which I don't know if you can perceive. It's just about perceivable.
SPEAKER_01It is perceivable, but like only if you want to be very scientific about it, right? I think in practice, it if it's really 300 and no more, then it's kind of immaterial for a real conversation. I mean, round trip times in a bad telephony setup can cost you more than that. Potentially. That's really interesting. And you know, I kind of like just putting like a bit of a marketing veneer on it. You could really say that it's the best dialogue model that can reason in under 300 milliseconds, or maybe like 0.5 if you put it all together right.
SPEAKER_00That way of framing it. Like we're not saying we've trained the best reasoning model, but we're saying this is the best one that's reasoning in real time.
SPEAKER_01The best real time reasoning model like in the world right now.
SPEAKER_00Yeah, for getting stuff done in a voice call. And I'll just like chat about latency and stuff. I would recommend listeners to check out the blog post, which should be out when this is released. We got some comparison versus like GPT Real Time, for example, where our medium latency, we're aiming for under 300 milliseconds, it's in like 280, 250 or something.
SPEAKER_01Okay, well, talking about evals then, how did we evaluate like its performance other than the latency piece where we were kind of like pretty uncompromising around like this limit?
SPEAKER_00We are releasing our internal benchmarks soon. Um, I think it's gonna be called Dialogue Aval. And there are results from a sort of early version of that in the blog post. And there you'll see that dialogue reason one is performing the best amongst the real-time models. We also show you know the big reasoning, slow models like Gemini Pro as an idea for the ceiling. Like, what if you were allowed to think for a minute before you replied on this dataset? What would you get?
SPEAKER_01Why was Gemini Pro chosen as the ceiling versus an anthropic model?
SPEAKER_00Or the Gemini Pro is one that supports audio on the input side, unlike the clawed ones. So it's a nice model in that uh you can feed it the conversation the same way you can dial reason one. Are there any other foundational ones which would be like contenders for that like L1 judge in? We have the GPT real-time models and then the various sort of open source models as well, which would require post-training to be competitive. We've taken an open waste model and post-trained it to do this. So all these open waste models they can hear and they can respond. Just uh they don't have this turn-taking capability and they don't have very reliable function calling.
SPEAKER_01I mean, look, function calling really is where DS2S models also fall over right now, right?
SPEAKER_00Was definitely the case six months ago, and they're getting better. But it is kind of a difficult task to mix speech with function calling, I think. And that it can be quite hard, I think, for these models to understand what agent framework they're in. Like, what happens if I call a function? Do I need to say something before I call a function? Will I get a chance to call another function? If I say something before I call this function, is that it? I'm not going to be allowed to. So, like part of the reason that we train a model is that all of that is just built into the weights, because we train it to work in our agent framework where we have answers to all of those questions. The model that would become allergic to you know speaking before calling a function, because it's if you speak in our framework, you're not allowed to call a function after.
SPEAKER_01It's fascinating.
SPEAKER_00Okay, like maybe total evals and kind of like things we measure. So one thing we pull out is this audio category. So those are examples in the data set where audio should help. So things like identify the background noise conditions to like in the context, there's a menu for a restaurant and the user's like mispronounced one of the things. So the first one, you definitely need to hear the audio to know the background noise. The second one, like it should certainly help to hear the audio and to have the context. Those types of examples, that's where you see the big jump. So I guess the point is that having an audio native model like not only improves the speech understanding, but also opens up new use cases. So you can prompt it in ways that you wouldn't have before. And that's what's measured in this audio category. So you could prompt it to be like, if the user is somewhere noisy, then tell them you're gonna do your best, but like suggest that they move somewhere quieter if possible. Or if it sounds like the user is struggling to get their customer number, you say, okay, we could also look you up by your postcode or something. Interrupt the user and say that like it's like new use cases that the cascaded mods they should be sort of random or educated guessing level. We actually did see that GPT Realtime on that category is about as the same as cascaded pipeline systems. So uh deserves a bit more looking into, but to me it suggests that this is a you know fancy pipeline system, like a well-engineered ASR-L MTTS rather than a true speech-to-speech model as advertised.
SPEAKER_01Yeah, I mean it's interesting. I think that kind of like that surfacing out kind of like puts into question like there's a whole like vibe feeling of a demo where you do one thing versus like what's really under the hood, and like how important it actually is. Because do you think it's more likely that it is like a semi-cascaded system, or rather like a safeguarded system that might have like cascaded bits intervening? Kind of like R V A D is there to provide like a signal.
SPEAKER_00I think there's pretty strong evidence from what I've seen that it's going through like a text bottleneck between speech and an LLM part. I had that sort of um demo I post on LinkedIn a while ago where you, if you ask it, do Americans pronounce the end of the alphabet XYZ? Then it responds, yes. Americans respond the end of the alphabet as XYZ, which is what you would get if it had been transcribed as just a letter XY in a character. So yeah, I mean that that is that's one anecdotal example. But obviously, on our dialogue about benchmark, we're going to be releasing soon. Uh, you can see for yourself, I suppose, and evaluate whatever models you like.
SPEAKER_01Why do we not use other things? You know, I think that you know, there are things like the Telbench that people talk about as like benchmarks for this. What was wrong with them?
SPEAKER_00I think well the yeah, Taubench has a couple that has the original sort of text-based one and the more recent voice version. And the text one is like, okay, what would happen if uh Fable was allowed to run a banking customer call? Okay, it's interesting, but it's not uh practical. The latency there would be crazy. And in reality, you know, we might use something like Fable to build the agent that talks to customers of a bank. But we're not going to use Fable to actually run thousands of calls a day with any with the customer. The voice one, it has unrealistic prompting in that I think it's a good thing to measure still. And like how well do these models do with quite open prompts and general tools that they're allowed to call. But it's not a targeted evaluation of can an LLM do all the things it needs to to make a customer call work. It relies a bunch on the user simulator being accurate, which it is not really, where there's a lot of variability there. And yeah, I think if you were to build a practical system, you'd not prompt it in the way that they have in the in this evaluation. So we're interested in that benchmark, but we wanted to build dialogue of Val so that has some realistic practical kind of prompts that you would see. And it's targeted in that we're not relying on the user simulator, but we're just putting you in all of the balanced context you would find yourself in if you're an LLM given this type of task, a targeted evaluation of all of those capabilities.
SPEAKER_01Okay. Well, look, I mean, I think evaluating dialogue systems has always been hard. I think we've released that we're gonna do our best to give the community something that they can use as a more like pragmatic real proxy for whether they've got a good model or not. I think the more exciting bit is also we've got dialogue region one in production in the platform right now. We've got it in the hands of a few select partners that are already kind of like putting it inside their platforms as well. It will be released as an API in the coming weeks. And if you're interested in trying it out early, you can get in touch with us and get into the early access plan as well. We're keen for feedback and for just usage because it's something we've worked on for a long time and really excited for the world to experience both in like using our platform and using other platforms. Super excited for more people to get their hands on it. Yeah. Well, Matt, thank you for today. More importantly, thank you for leading this work and getting us here. And as always, guys, please like, share, subscribe, and we'll see you in the next one. Bye. End of turn. End of turn.
