An ASR to LLM to TTS voice agent on LiveKit, serving browser and phone callers from one worker. Built to find out where the latency actually goes, and to make every vendor a config line rather than a rewrite.
How it works
A caller joins a LiveKit room, from a browser over WebRTC or from a phone through a SIP trunk. The agent code does not branch on which; the SIP bridge turns the carrier leg into an ordinary room participant. What changes is the audio profile: codec, noise filter and interruption thresholds follow the path automatically.
Audio passes through a noise filter that isolates the primary speaker, then a voice activity detector and a contextual end-of-turn model that reads the audio and the conversation together. That is what allows a short silence timeout without cutting someone off mid-thought. A plain VAD timeout cannot tell 'I want a flight to' followed by a pause from a finished sentence.
The transcript streams into the LLM, the reply streams into TTS, and the agent starts speaking on the first audio chunk. Every stage is timed per turn and pushed to a dashboard while the call is in progress.
Measured end to end on gpt-4.1, no downgrade to a smaller model: 902 ms mean over eight real turns, fastest 552 ms, slowest 1216 ms. The single biggest win was the ASR. Swapping OpenAI transcription for Deepgram Flux took the same pipeline from 2089 ms to 902 ms, because Flux signals end of turn early enough for the LLM to start on the partial transcript.
Key features
- One worker serves WebRTC and telephony callers; the agent never branches on transport
- STT, LLM and TTS are each one environment variable; any stage can point at a self-hosted OpenAI-compatible server
- Contextual end-of-turn detection, barge-in handling, backchannel tolerance and false-interruption resume
- Per-turn stage timings streamed to a live dashboard, phone and browser calls side by side
- Simulation suite where a scripted caller actually speaks, so turn taking is tested with real audio and fails CI when it regresses
- Noise cancellation profiles for wideband web audio and 8 kHz carrier audio
Step by step
- Caller joinsBrowser via WebRTC, or phone via a SIP trunk that lands in the same room.
- Noise filterBackground voice cancellation isolates the primary speaker. A telephony variant handles 8 kHz carrier audio.
- Turn detectionSilero VAD plus a contextual end-of-turn model. Floor 300 ms of dead air, ceiling 2.5 s.
- ASRDeepgram Flux streams the transcript and signals end of turn early, which is what makes preemptive generation useful.
- LLMgpt-4.1 starts on the partial transcript. First token at 594 ms.
- TTSCartesia returns first audio at 204 ms, and the agent starts speaking on the first chunk.
- MeasureEach turn's stage timings are written to a metrics file and pushed to the dashboard over the room data channel.
What the simulation showed
Five scripted scenarios, each a caller that really speaks into the room. Three pass, and they are the ones about behaviour: the agent does not cut in on a mid-sentence pause, a real interruption stops it, and an 'mm-hm' does not.
The two failures are the latency assertion measured caller side, which includes both network legs and so reads above the agent's own figure. That is the number the person on the phone feels, so it stays as a gate.