What is an Audio Decision Model?

It isn't a transcriber and it isn't a chatbot. An audio decision model listens to a conversation and answers questions about it, fast, with a probability attached. Here's what that means and where it's useful.

An audio decision model listens to audio and answers a question about it. Is the caller done talking? Which language are they speaking? Did they just read out their date of birth? The answer comes back as a choice with a probability attached, fast enough to act on while the conversation is still happening.

If you build voice AI, your agent already makes dozens of these calls on every conversation, usually with a pile of separate tools: a silence timer here, a regex there, an LLM prompt for the rest. This post explains what an audio decision model is, how it differs from the models you already use, and the decisions it can take off your hands.

And it gives one thing back: an answer, in one of three shapes.

Every answer comes with a probability, so your code decides how sure is sure enough.

The important part is that the question is an input. Most audio models are trained to answer one fixed question. A voice activity detector (VAD) only knows speech from not-speech. An emotion classifier only knows emotions. An audio decision model takes the question at request time, so the same model can answer the turn-taking question, the language question and the frustration question about the same second of audio. A new decision is a new question, not a new model.

Most voice stacks turn speech into text first and make every decision on the text. That throws away most of what a decision needs.

It doesn’t write replies, and it doesn’t replace your LLM. It sits next to your stack and handles the small, frequent, time-critical calls that the LLM is too slow for and can’t hear. We wrote about why that layer is missing from most stacks in the missing piece in the voice AI stack.

Short answer: anything you can phrase as a question about the audio with a short list of answers. In a voice agent, those questions fall into six groups.

There’s more on these in Ask what the moment needs.

An LLM decides which tool to call. It can’t tell you when the caller has said enough to call it, or whether “yeah” was a real yes. An audio decision model can, for each kind of tool:

The same model works on recordings, where speed matters less and coverage matters more.

On a real call, these questions don’t arrive one at a time. Here’s a 30-second call about a late order, with every decision a voice agent has to make along the way.

Nine decisions in 30 seconds, and most of them happen while someone is still talking. Today they’re spread across a VAD, a silence timer, an answering machine detector, an LLM prompt and a regex for card numbers. Each was tuned on its own, so sooner or later they disagree. Ask one model the questions about the same audio, and they share one view of the conversation.

Not every mistake costs the same. Masking a number that wasn’t sensitive costs nothing. Refunding an order on a mumbled “yeah” costs money. So each decision needs its own bar.

That only works if the probabilities are calibrated: when the model says 0.8, it should be right about 80% of the time. Then a threshold means something, and you can move it per action, per customer or per use case without retraining anything.

That’s the whole idea: audio in, your question in, an answer you can act on. It’s also what we’ve been building at Tazo.

A transcript tells you what was said. An audio decision model tells you what to do about it.