Voice AI in Contact Centers: STT/TTS Architecture for Hebrew and Arabic

Why English-first stacks break in Hebrew and Arabic

Most off-the-shelf voice AI pipelines are tuned for English. Hebrew and Arabic break them in four predictable ways:

• Right-to-left text propagates display bugs into transcripts, dashboards and downstream LLMs.
• Diacritics (Hebrew niqqud, Arabic tashkeel) are usually missing in input but required for accurate TTS prosody.
• Code-switching — speakers mix Hebrew/Arabic with English brand names mid-sentence; monolingual models silently mistranscribe.
• Dialect variance — modern colloquial Hebrew differs from formal Hebrew; Arabic varies dramatically between MSA and Levantine, Egyptian, Gulf, Maghrebi dialects.

The latency budget for a natural-sounding call

A natural phone conversation tolerates roughly 700ms round-trip before it feels robotic. In our deployments:

Carrier + SBC + media bridge      ~120 ms
Streaming STT (partial → final)   ~150 ms
LLM first token                    ~200 ms
Streaming TTS first chunk          ~150 ms
Network return                     ~80 ms
                                  ──────
Total                              ~700 ms

Every millisecond above 700 increases caller dropout. The single biggest lever is colocating STT, LLM and TTS workers on the same network as the telephony layer.

The architecture pattern we deploy

1. Telephony layer — self-managed SIP stack for sub-200ms media path.
2. Streaming STT — partial-results pipeline so the LLM can start "thinking" before the speaker finishes.
3. Language router — a small classifier flags HE / AR / EN per utterance to pick the right downstream model.
4. LLM tier — function-calling model with structured outputs for CRM updates and ticket creation.
5. Streaming TTS — chunked synthesis so audio starts playing within ~150ms of the first token.
6. Barge-in handling — the caller can interrupt; the agent stops, recovers context and continues.

What we measure in production

• Word Error Rate (WER) per language and per dialect, not just overall.
• Time-to-first-audio-byte — the metric callers actually feel.
• Barge-in recovery rate — how often the agent gracefully picks up after being interrupted.
• Cost per resolved call, not per minute.

Field notes from deployments

Two patterns we see repeatedly:

• Customers always overestimate how often callers stay in one language. Plan for code-switching from day one.
• The single largest quality win is usually not a better model — it is tighter colocation between telephony and inference.