AI Voice Training Simulator for Call Agents
Problem
Companies whose staff spend their day on the phone — sales, support, collections — need a way to bring new agents up to speed and keep experienced ones sharp. Traditional training leans on role-play with a manager or colleague: it ties up a senior person for every session, can't run on demand, and the feedback is subjective and inconsistent from one trainer to the next. Practising on real customers, meanwhile, means making mistakes on live calls. Our client needed a way for consultants to rehearse realistic calls as often as they liked, in a safe environment, and to receive objective feedback on how they did.
Result
Agents can now practise lifelike calls on demand, as many times as they need, without a trainer in the room or a real customer on the line. Every session ends with objective, structured feedback — a grade, specific comments, and a report — so trainees see exactly what to improve and managers get a consistent, comparable measure of readiness across the whole team. The result is faster, more uniform onboarding and a repeatable way to keep skills current, with none of the risk of learning on real customers.
How did we achieve it?
We built a real-time voice pipeline that role-plays a client on a live call. A voice-activity-detection (VAD) model listens for when the trainee is speaking; once they finish, an ASR model transcribes the speech to text and passes it to an LLM that plays the client — interpreting what was said and answering with its own questions and objections. That reply is spoken back to the trainee through a TTS model, so the whole exchange feels like a natural conversation rather than typing. The call runs until the trainee completes the task or the simulated client ends it. A separate evaluation model then reviews the entire dialogue, pinpoints where the trainee went wrong, assigns a grade, attaches comments, and generates a report. Keeping latency low across the VAD → ASR → LLM → TTS loop was central to making the conversation feel real.
How a practice call works
- Detect speech. A VAD model recognises when the trainee is talking and when they've finished their turn.
- Transcribe. An ASR model converts what the trainee said into text.
- Respond as the client. An LLM interprets the text and replies in character, asking questions and raising objections like a real customer.
- Speak back. A TTS model voices the LLM's reply, so the trainee hears a natural conversation.
- End & evaluate. When the task is completed or the client ends the call, a separate model analyses the full dialogue, grades it, adds comments, and produces a feedback report.