CODESAGEZ/INCIDENT POST-MORTEMS

Edge Cases & Root Cause Analysis

Production engineering audit detailing real failure modes encountered during repository AST ingestion, QLoRA quantization, and vLLM continuous batching, with applied mitigations.

INC-01

Catastrophic Forgetting of General Capabilities

SEV-1 // CRITICAL
[ FAILURE TRIGGER ]

After fine-tuning on code, the model loses performance on general NLP tasks. A model that can code but fails to understand a long code review request is useless.

[ ROOT CAUSE DIAGNOSIS ]

Gradient updates during SFT push weights toward code-specific patterns. With 3 epochs, weights drift far from pre-trained values.

> APPLIED ARCHITECTURAL MITIGATION:
Mixed dataset: 70% code + 30% general instruction pairs. Use instruction-tuned base (not base model). Monitor eval_loss on general instruction eval set.
Residual Risk Boundary: ~5-8% regression on MMLU vs base instruct model. Acceptable for code-specialized tool.[STATUS: STABILIZED]
INC-02

QLoRA Quantization Noise on Sensitive Layers

SEV-2 // ELEVATED
[ FAILURE TRIGGER ]

4-bit NF4 quantization introduces rounding errors that accumulate in first and last transformer layers. Precise token probabilities (choosing between = and ==) show measurable error rate increase.

[ ROOT CAUSE DIAGNOSIS ]

NF4 maps fp16 values to nearest of 16 discrete values. In early layers, this error propagates through all subsequent layers.

> APPLIED ARCHITECTURAL MITIGATION:
Keep lm_head in fp16. Use double quantization. After merging: run INT8 calibration pass (AutoGPTQ) for inference.
Residual Risk Boundary: ~1-2% accuracy regression vs fp16 serving. Production vLLM serves in BF16 post-merge.[STATUS: STABILIZED]
INC-03

vLLM Memory OOM at High Concurrency

SEV-1 // CRITICAL
[ FAILURE TRIGGER ]

At 32+ concurrent requests with long context (4K+ tokens), PagedAttention block allocator runs out of GPU memory. All in-flight requests affected simultaneously.

[ ROOT CAUSE DIAGNOSIS ]

PagedAttention allocates blocks on demand. Under memory pressure, vLLM swaps to CPU RAM. At extreme concurrency, swap bandwidth becomes bottleneck.

> APPLIED ARCHITECTURAL MITIGATION:
gpu_memory_utilization=0.90. max_num_seqs=32. max_model_len=8192. Circuit breaker: if p99 > 5s, shed to CodeSage-mini fallback.
Residual Risk Boundary: Burst traffic can exhaust queue. Modal auto-scales but cold start takes 18-25s.[STATUS: STABILIZED]
INC-04

Overconfident on Out-of-Domain Code

SEV-2 // ELEVATED
[ FAILURE TRIGGER ]

Trained primarily on Python/TS/Rust/SQL. For Kotlin, Haskell, or Solidity, generates syntactically plausible but subtly wrong code.

[ ROOT CAUSE DIAGNOSIS ]

Fine-tuning increases confidence on in-domain distributions. Calibration for uncertainty on OOD inputs degrades.

> APPLIED ARCHITECTURAL MITIGATION:
Logprob threshold: if mean token confidence < 0.7, route to base model. Language detection in preprocessing. System prompt includes uncertainty instruction.
Residual Risk Boundary: Calibration catches ~70% of confident-but-wrong cases. 30% slip through.[STATUS: STABILIZED]
INC-05

Adapter Merge Changes Output Distribution

SEV-3 // LOW
[ FAILURE TRIGGER ]

After merge_and_unload(), merged model produces slightly different outputs than pre-merge. Benchmark numbers may not match.

[ ROOT CAUSE DIAGNOSIS ]

BitsAndBytes 4-bit dequantization introduces small errors during merge. Errors compound across generation process.

> APPLIED ARCHITECTURAL MITIGATION:
Always evaluate AFTER merge. Cross-check 100 samples pre/post merge. Prefer fp16 merge then quantize separately.
Residual Risk Boundary: ~0.2-0.5pp benchmark difference. Accepted as measurement noise.[STATUS: STABILIZED]
INC-06

vLLM Output Non-Determinism

SEV-3 // LOW
[ FAILURE TRIGGER ]

At temperature=0, different concurrency levels produce different outputs. Continuous batching changes which attention computations are fused together.

[ ROOT CAUSE DIAGNOSIS ]

GPU floating-point operations are not strictly commutative. Fused attention kernels produce different rounding results depending on batch composition.

> APPLIED ARCHITECTURAL MITIGATION:
For code completion: acceptable (valid alternatives are valid). For tests: use seed parameter. Document: exact determinism not guaranteed at high concurrency.
Residual Risk Boundary: No full fix without disabling continuous batching. 24x throughput trade-off is worth it.[STATUS: STABILIZED]