LLM Cost Optimization at the Orchestration Layer

Stop optimizing prompts. Start optimizing the orchestrator.

Most teams try to cut LLM costs by editing prompts. The bigger wins live one layer up — at the orchestration plane that decides which model runs, when, and with how much context.

Seven techniques, in order of impact

1. Model routing — classify the task first with a cheap model, then route to the cheapest model that meets the quality bar. Frontier models for <20% of calls.
2. Semantic caching — cache responses by embedding similarity, not exact string match. Hit rates of 30–50% are normal for support workloads.
3. Context compression — replace raw chat history with a rolling summary written by a small model. Cuts input tokens 5–10×.
4. Speculative decoding at the provider level — opt in where supported.
5. Per-agent budgets — hard ceilings enforced by the scheduler, not application code. An agent that exceeds its budget is killed, not throttled.
6. Batch where possible — embeddings, classifications and evals on batch endpoints at 50% of synchronous price.
7. Cold storage for context — push old episodic memory into a vector store, not into every prompt.

What this looks like in production

On a typical 1M-call/month contact center workload, applying all seven cuts blended cost-per-call from $0.084 to $0.027 — a 68% reduction — with no measurable quality drop on a 500-ticket eval set.

Why the orchestrator is the right place

Routing, caching and budgets are cross-cutting. Implement them per agent and you re-implement them inconsistently. Implement them in the orchestration layer and every agent inherits them for free. That's the architectural argument for platforms like AioClaw over hand-rolled framework code.