Phase 2: High-Performance Serving

Inference Architecture

Deep dive into the vLLM serving infrastructure that enables 12,500 tokens/second throughput and 24x the performance of naive HuggingFace pipelines.

Request Router

vLLM OpenAI-compatible API endpoint

vLLM exposes /v1/completions and /v1/chat/completions API identical to OpenAI spec. No code changes needed to switch from OpenAI API to self-hosted CodeSage. Request validation, rate limiting, priority queue for MCP tool calls.

Continuous Batching Scheduler

Dynamic request batching for maximum GPU utilization

Traditional static batching: wait for N requests, process, repeat. vLLM continuous batching: as soon as any sequence finishes, a new request joins immediately. GPU never idles. Stable latency until ~100-150 concurrent requests.

PagedAttention

THE core innovation — OS-style virtual memory for KV cache

Traditional serving pre-allocates contiguous GPU memory at MAX sequence length. 60-80% of GPU RAM wasted. PagedAttention: KV cache divided into fixed-size blocks (16 tokens), allocated on-demand, non-contiguous. Memory waste: <4%. Throughput: 2-24x vs naive serving.

Model Workers

Tensor parallel model execution

For 8B: single GPU sufficient. Tensor parallelism splits attention heads across GPUs. PagedAttention shares prompt KV blocks across sequences. Copy-on-write at block granularity. vLLM V1 engine: incremental state updates, near-free prefix caching.

Output Streaming

Token-by-token streaming via Server-Sent Events

SSE streaming like ChatGPT. TTFT: 72ms at low concurrency (H100). Inter-token latency: ~8ms/token. A 200-token completion: 72ms + 200×8ms = 1.7s. User perceives streaming output starting at 72ms.