CODESAGEZ/INFERENCE INFRASTRUCTURE

vLLM Inference Engine & PagedAttention

High-throughput serving architecture powered by PagedAttention and continuous batching. Eliminates KV cache memory fragmentation for sub-100ms TTFT.

KV Cache Memory Efficiency (2048 Context Length)

MEMORY REDUCTION: 76%
Standard Serving (Contiguous Pre-allocation)68% Wastage

Pre-allocates memory for maximum sequence length (8k). Unused slots cannot be shared across concurrent queries.

PagedAttention (Dynamic 16-Token Blocks)<4% Wastage

Allocates KV cache into fixed 16-token virtual pages on demand. Zero internal fragmentation; 2.4x higher throughput.

Serving Subsystems & Execution Latency

ENGINE: vLLM V1
01
Request Router
vLLM OpenAI-compatible API endpoint
vLLM exposes /v1/completions and /v1/chat/completions API identical to OpenAI spec. No code changes needed to switch from OpenAI API to self-hosted CodeSage. Request validation, rate limiting, priority queue for MCP tool calls.
02
Continuous Batching Scheduler
Dynamic request batching for maximum GPU utilization
Traditional static batching: wait for N requests, process, repeat. vLLM continuous batching: as soon as any sequence finishes, a new request joins immediately. GPU never idles. Stable latency until ~100-150 concurrent requests.
03
PagedAttention
THE core innovation — OS-style virtual memory for KV cache
Traditional serving pre-allocates contiguous GPU memory at MAX sequence length. 60-80% of GPU RAM wasted. PagedAttention: KV cache divided into fixed-size blocks (16 tokens), allocated on-demand, non-contiguous. Memory waste: <4%. Throughput: 2-24x vs naive serving.
04
Model Workers
Tensor parallel model execution
For 8B: single GPU sufficient. Tensor parallelism splits attention heads across GPUs. PagedAttention shares prompt KV blocks across sequences. Copy-on-write at block granularity. vLLM V1 engine: incremental state updates, near-free prefix caching.
05
Output Streaming
Token-by-token streaming via Server-Sent Events
SSE streaming like ChatGPT. TTFT: 72ms at low concurrency (H100). Inter-token latency: ~8ms/token. A 200-token completion: 72ms + 200×8ms = 1.7s. User perceives streaming output starting at 72ms.

End-to-End Latency Profile (200 Output Tokens)

TTFT (Time To First Token)
72 ms
Prefill phase (H100)
Inter-Token Latency
8 ms/tok
Continuous decode batching
Total Ingestion & Generation
1.67 s
Complete RAG answer delivery