GPU, VRAM and Cost: Hardware Optimization for AI Systems at Scale

The three numbers that decide your GPU bill

1. Model footprint — weights + KV cache at your target context length.
2. Concurrency — how many requests share a single GPU before tail latency degrades.
3. Duty cycle — what fraction of the hour the GPU is actually busy.

Get any one of these wrong and your unit economics collapse.

VRAM sizing rules we use

• For a 7B model at 8k context: 16–20 GB VRAM with quantization, 28–32 GB at FP16.
• For a 70B model: 2× A100 80GB minimum, or tensor-parallel across 4× L40S.
• Always reserve ~15% of VRAM for KV-cache spikes — running at 100% utilization causes OOM under burst load.

Load balancing AWS and RunPod

We split workloads by characteristic:

• AWS for steady-state, low-latency, latency-SLA workloads — predictable, integrated with our VPC and observability.
• RunPod / spot GPU clouds for batch jobs: lead scoring, embeddings, nightly re-ranking, video analysis.

A simple queue + capability tag in AioClaw routes each task to the cheapest GPU pool that meets its latency requirement.

Cost-benefit: scoring 500,000 leads

A real workload we ran for a B2B exporter:

Input:       500,000 leads × ~1,200 tokens enrichment
Pipeline:    embeddings → classifier → LLM scorer → human review queue
On-demand:   ~$4,800 (AWS g5.2xlarge, 18h)
Spot batch:  ~$640  (RunPod A40, 22h)
Quality:     identical (the human review set was the same 4%)

The lesson is not "always use spot" — it is "match the workload to the right hardware and let orchestration handle the routing."

What we measure in production

• Tokens/sec per GPU per model.
• Queue depth and waiting time per provider.
• Cost per resolved task (not per call, not per token).