AI Telephony Infrastructure: Why We Run Our Own Stack

The latency budget of a real-time AI call

A natural phone conversation tolerates roughly 700ms of round-trip latency before it starts to feel robotic. That budget has to cover: audio capture, VAD, streaming STT, LLM inference, TTS, and the return trip over the carrier. Off-the-shelf cloud telephony providers consume 300–500ms of that budget before your model has even seen a token.

Why we moved to a self-managed Asterisk stack

AIO Call Center runs on an independent, Asterisk-based telephony layer co-located with the AI inference workers. The benefits compound:

• Sub-200ms media path between the carrier and the AI worker.
• Streaming barge-in — the caller can interrupt the bot mid-sentence and the agent recovers gracefully.
• Per-call observability — every leg of the SIP dialog, every model token and every tool call goes into one trace.
• Cost control — we pay for trunks and compute, not per-minute markups on AI features.

Architecture in one diagram

Carrier ─► SBC ─► Asterisk ─► Media bridge ─► AI worker
                                              │
                                              ├─► Streaming STT
                                              ├─► LLM (function calling)
                                              └─► Streaming TTS

Edge cases that justify the investment

• High concurrency campaigns — thousands of simultaneous outbound calls without provider throttling.
• Automated QA — every call is scored against a rubric by a second LLM before a human ever listens.
• Regulated industries — recordings, transcripts and consent live on infrastructure we control.

When NOT to self-host

If your call volume is low and latency is not critical, a managed CPaaS is still the right answer. The break-even for our customers tends to sit around 20,000 minutes/month of AI-driven calls.