Realtime voice agents over telephony: what actually breaks
Speech-to-speech models demo beautifully in a browser with a good microphone. Put the same model on a real phone line and a different project begins. Having spent time on the telephony side of realtime audio systems, these are the failure modes I would warn any team about before they write the first line of code.
The latency budget is spent before the model sees anything
A conversational turn feels natural somewhere under a second. That budget is consumed by pieces most teams forget to count: the carrier media leg, the jitter buffer, voice activity detection deciding the caller has finished, the model's time to first audio, and playback buffering on the way back. The model's inference latency is often the smallest slice.
The tempting knob is the VAD silence threshold: shorten it and the agent responds faster. But the tradeoff inverts hard. Trim too aggressively and the agent starts answering people who were only pausing to think, and clipping the first syllable of fast speakers. One of the few teams publishing production numbers on this, Canarics, a Baltic voice-AI vendor, reports cutting roughly 250ms per turn through silence-detection and padding tuning, and concluding the next 250ms is not worth chasing: the floor is set by human pause behavior, not by compute. That matches my experience. Complaints about being talked over are your regression test.
Barge-in means fighting your own echo
Callers interrupt. The agent must stop talking within a few hundred milliseconds or it feels like an IVR. But the moment you implement interruption, speakerphones start interrupting the agent with its own voice: the agent's audio leaks from the caller's speaker into their microphone and arrives back as "caller speech". Without echo cancellation on your side of the line, the agent hears itself, stops, and stutters. Carrier-side AEC exists but is inconsistent; assume you need your own.
Narrowband audio is not your main quality problem
Telephony largely still means 8 kHz narrowband codecs. The instinct is to blame lost words on the pipe and pursue wideband end to end. The published evidence says otherwise: Canarics rebuilt their audio path for lossless wideband delivery and found capture did not improve, with 6 to 8% of short utterances ("yes", a name, a number said quickly) still missed. That is the current floor of realtime speech models, not of the plumbing.
The engineering consequence is a design rule, not a codec choice: the conversation must survive losses. The agent asks again naturally instead of guessing, and nothing gets written to a database from a half-heard word. Grounding rules like "the caller's own confirmed words are the only source for structured data" belong in the system design from day one.
Answering machine detection lies about one call in eleven
For outbound calling, carrier answering-machine detection sounds like a solved problem. Measured false-positive rates say otherwise: Canarics reports 8.7% of "machine" verdicts are actually live humans, typically slow speakers, noisy environments, or unusual greetings. An agent that hangs up on the carrier's verdict hangs up on real people at scale. The pattern that works: treat the detector as a hint, verify conversationally, and once a human has spoken, no automated signal may end the call.
Multilingual callers do not pick a language
In multilingual markets, callers switch languages mid-sentence. The same published dataset measures 6% of callers crossing the Cyrillic-Latin script boundary within one conversation, and that undercounts drift between Latin-script languages. Any design that asks the caller to select a language up front, or pins the model to one locale, fails on exactly the turn where it matters. Language must be free to follow the caller.
The model is the smallest problem
The consistent lesson from teams operating these systems, and from my own work around them: the model is the easy part. The real system is everything around it, deciding what happens on a mishear, who may end a call, what reaches the CRM and on whose authority. Demos test none of that. Traffic tests all of it. If you are evaluating this space, favor whoever publishes production numbers with sample sizes; measured call statistics are still rare enough to be a signal by themselves.