Phase 3: Post-Mortems

Failure Modes & Fixes

Building LLM systems is messy. This is a transparent log of what went wrong during training and deployment, and the engineering required to fix it.

Catastrophic Forgetting of General Capabilities

high SEVERITY

The Problem

After fine-tuning on code, the model loses performance on general NLP tasks. A model that can code but fails to understand a long code review request is useless.

Root Cause

Gradient updates during SFT push weights toward code-specific patterns. With 3 epochs, weights drift far from pre-trained values.

The Fix

Mixed dataset: 70% code + 30% general instruction pairs. Use instruction-tuned base (not base model). Monitor eval_loss on general instruction eval set.
Residual Risk: ~5-8% regression on MMLU vs base instruct model. Acceptable for code-specialized tool.

QLoRA Quantization Noise on Sensitive Layers

medium SEVERITY

The Problem

4-bit NF4 quantization introduces rounding errors that accumulate in first and last transformer layers. Precise token probabilities (choosing between = and ==) show measurable error rate increase.

Root Cause

NF4 maps fp16 values to nearest of 16 discrete values. In early layers, this error propagates through all subsequent layers.

The Fix

Keep lm_head in fp16. Use double quantization. After merging: run INT8 calibration pass (AutoGPTQ) for inference.
Residual Risk: ~1-2% accuracy regression vs fp16 serving. Production vLLM serves in BF16 post-merge.

vLLM Memory OOM at High Concurrency

high SEVERITY

The Problem

At 32+ concurrent requests with long context (4K+ tokens), PagedAttention block allocator runs out of GPU memory. All in-flight requests affected simultaneously.

Root Cause

PagedAttention allocates blocks on demand. Under memory pressure, vLLM swaps to CPU RAM. At extreme concurrency, swap bandwidth becomes bottleneck.

The Fix

gpu_memory_utilization=0.90. max_num_seqs=32. max_model_len=8192. Circuit breaker: if p99 > 5s, shed to CodeSage-mini fallback.
Residual Risk: Burst traffic can exhaust queue. Modal auto-scales but cold start takes 18-25s.

Overconfident on Out-of-Domain Code

medium SEVERITY

The Problem

Trained primarily on Python/TS/Rust/SQL. For Kotlin, Haskell, or Solidity, generates syntactically plausible but subtly wrong code.

Root Cause

Fine-tuning increases confidence on in-domain distributions. Calibration for uncertainty on OOD inputs degrades.

The Fix

Logprob threshold: if mean token confidence < 0.7, route to base model. Language detection in preprocessing. System prompt includes uncertainty instruction.
Residual Risk: Calibration catches ~70% of confident-but-wrong cases. 30% slip through.

Adapter Merge Changes Output Distribution

low SEVERITY

The Problem

After merge_and_unload(), merged model produces slightly different outputs than pre-merge. Benchmark numbers may not match.

Root Cause

BitsAndBytes 4-bit dequantization introduces small errors during merge. Errors compound across generation process.

The Fix

Always evaluate AFTER merge. Cross-check 100 samples pre/post merge. Prefer fp16 merge then quantize separately.
Residual Risk: ~0.2-0.5pp benchmark difference. Accepted as measurement noise.

vLLM Output Non-Determinism

low SEVERITY

The Problem

At temperature=0, different concurrency levels produce different outputs. Continuous batching changes which attention computations are fused together.

Root Cause

GPU floating-point operations are not strictly commutative. Fused attention kernels produce different rounding results depending on batch composition.

The Fix

For code completion: acceptable (valid alternatives are valid). For tests: use seed parameter. Document: exact determinism not guaranteed at high concurrency.
Residual Risk: No full fix without disabling continuous batching. 24x throughput trade-off is worth it.