Hover a node · click to focus · ←/→ steps
Apps → tagged gateway → AWS / Azure / GCP providers or self-host. Cost tags feed FinOps; sampled traffic closes the eval loop.
When you need this
- Need EU data residency for prompts/completions
- API spend dominates; open weights are viable
- Latency to public APIs misses SLOs
- Want continuous batching on owned GPUs
Prerequisites
- GPU nodes with capacity plan
- Model license review completed
- LiteLLM (or gateway) for routing already planned
Tools
Assess TCO vs managed APIs; needs GPU ops maturity and autoscaling discipline.
Deploy as single ingress before adding a second LLM vendor.
Assess TCO vs managed endpoints unless you already run K8s at scale.
Steps
- 1
Select model and hardware profile
Match context length, quantization, and GPU memory. Document tokens/sec target and max concurrent requests.
- 2
Deploy vLLM with OpenAI API shape
Expose the compatible endpoint behind internal ingress only. Enable continuous batching; set max_model_len intentionally.
- 3
Put LiteLLM in front
Apps never call vLLM directly — gateway applies tags, budgets, fallbacks to cloud models, and audit logs.
- 4
Autoscaling and GPU FinOps
Scale on queue depth / GPU util. Label pods for Kubecost. Prefer scale-to-zero or schedule for non-prod.
- 5
Eval parity before cutover
Run the same golden set against cloud and vLLM paths. Cut traffic only when quality and latency SLOs hold.
Adoption pitfalls
- Skipping gateway → no FinOps or policy
- Oversized context length wastes VRAM
- No cold-start plan for autoscaled replicas
Adoption checklist
- Internal-only ingress; no public model API
- All traffic via gateway with tags
- Golden-set parity vs previous provider
- GPU showback labels present
- Rollback to cloud model path tested
SEER REAL assessment / sprint
Assessment weighs sovereign inference vs API cost. Sprint stands up vLLM for one model behind LiteLLM with eval parity and rollback.