powered voice agent platform providerPhonely released Alma, an LLM built for voice agents. No Jitter spoke with Phonely CEO and co-founder Will Bodewes for insight into the business problem Phonely seeks to address, how Alma works and some of the common challenges contact centers face when rolling out voice AI agents
This conversation was lightly edited for length and brevity.
NJ: The announcement stated that a model trained on text is more prone to stumbling during a live conversation. Why?
Bodewes:Most of the industry runs onGPT 4.1, which is a model[OpenAI is] discontinuing. The reason why most of the industry runs on GPT 4.1 is because after GPT 4.1, OpenAI and other frontier labs started pushing in a different direction — making agents more long-running,better at coding, better at searching the entire internet, and they didn’t focus onlatency and conversational quality. And so, GPT 4.1 is not great, but it’s the best that we had gotten to at that point, which is a mix of latency and conversational quality.
We trained [our model] on how people talk and that’s going to make a significant difference. You don’t respond the same way over text as you do over the phone, and it turns out that when you train a model that way, you get better conversions and it sounds more natural.
NJ: If it’s a customer service call, how does the model handle multiple requests from the customer? Not necessarily on the voice side, but the “brain” — the LLM — has to know what’s going on, right?
Bodewes:Exactly. Part of what we built and what our whole platform is designed around is how you program the brain as easily as possible so that when you say “I want to schedule an appointment” and then “I want to check the pricing,” for example, it can actually understand those things. It will be able to follow the process and the steps to be able to schedule the appointment and then, if it’s a sales use case, it might ask you a question before it gives you the pricing. It depends on how the customer programs it.
We built the platform so that people have the brain and we built the model because we realized that all the models out there weren’t designed for what we use it for and what voice AI as a whole uses it [for], which is talking like a person while being able to handle the business logic that’s required. That’s the problem this solves. And it turns out that [Alma’s] both faster and cheaper and better than OpenAI and all the frontier labs on this specific use case.
NJ: And why is Phonely’s model faster, cheaper, better?
Bodewes:It’s faster because it’s asmaller model, and also cheaper because it’s a smaller model, and we can run it on our own hardware. What most people don’t know [about] is the latency you get from OpenAI. If you were to hit the OpenAI endpoint 60 times in a minute, so every second you’re polling it, sometimes that latency will spike —and it can spike to four, five, 10 seconds. We’ve seen 30 seconds before that response comes back, which means that if it’s a phone call, you make a request on the LLM and you finish, [but] you wait 30 seconds before it comes back.
The problem [here] is that it’s a queuing issue. Because OpenAI has a lot of people that are coming in, they’re constantly trying to under-provision their resources. When your request arrives, you arrive in a line. Sometimes you end up in the front of the line and sometimes you end up in the back, so you have to wait for all the other requests to process.
But for us, we have control, so we can run our own model on our own inference, and we can deal with this proactively. We can spin up multiple different regions, and we have control of the queue, whereas [with] OpenAI and the frontier labs, we don’t have control of the queue.
NJ: What are some of the challenges a call center might face scaling voice AI agents?
Bodewes:Number one, there’s latency issues, which this solves on. But number two, there’s instruction-following issues, which LLMs are great [at], except if you’ve ever tried to get ChatGPT to do anything, you know that sometimes it lies to you, sometimes it does the wrong thing and a lot of times it’s not repeatable and reliable.
Businesses need to have some level of conversational quality and repeatability. Marrying those two things is what makes a model valuable. It needs to follow the business logic every single time, like a robot, but it also needs to handle objections like a person and respond differently and have the natural conversational flow like a person. Those are the biggest problems that every business faces.
Once they get integrations and connections into their systems or software, all the problems arise from the LLM not following instructions when it should, not doing what it’s supposed to do and not talking like a person. [Alma’s] designed to fix those problems, basically.
NJ: Phonely’s model is still a large language model. It’s still non-deterministic. So, how are you fixing that deterministic piece?
Bodewes:It’s a hybrid. It’s trained on four different models so that gives it the deterministic architecture. But four different LLMs in itself doesn’t make it any smarter, right? It just means that you have four children at work.
Basically, [we’ve] designed our architecture so that one LLM is checking whether or not you asked the questions, another LLM is asking the questions and having the conversation, and another is extracting the information from it. You structure it in a graph, and then it basically throws a flag of yes or no. Did it accomplish the task? Yes or no, etc. One LLM is judging that, another LLM is running the conversation, and another LLM is extracting information and then we have one that’s checking the orchestrator.
You don’t get to 100% deterministic, but you increase the chance of determinism because what you have are specialized LLMs focused on specific parts of the problem.
NJ: Is there anything that your customers routinely forget to ask about when they pilot or go into production that might be useful for others to learn from?
Bodewes:I think they forget that AI is like a person — It needs to be trained and it needs to have instructions and it will take time for it to learn. We’ve been getting better about this as companies, but previously a lot of businesses were like, “I’m going to sprinkle some AI magic on it, and then it’s going to automatically work for my business. Realistically, that’s not the case.
You need to approach it just like you would approach hiring a new employee, but this employee will scale to millions of concurrent calls. But I still need to train that employee and coach it. The way I teach an AI employee is different than the way that I teach a human employee. They’re going to make different mistakes, but they’re both going to make mistakes. The mistakes [often] come from you forgetting that “yeah, I do ask this question” or “most people ask this question,” and there’s some contextual knowledge that I need to give it. It takes training and iteration.
During the sales process, we walk all of our customers through this, but it’s something that everyone should just think about as they choose different voice AI vendors. If a vendor does things right, they have the tools for you to train the [voice agents], and that’s what you should be evaluating: Does the vendor have the tools to train this AI agent and evaluate if they’re doing a good job?
