Edge Cases & Root Cause Analysis
Production engineering audit detailing real failure modes encountered during repository AST ingestion, QLoRA quantization, and vLLM continuous batching, with applied mitigations.
Catastrophic Forgetting of General Capabilities
After fine-tuning on code, the model loses performance on general NLP tasks. A model that can code but fails to understand a long code review request is useless.
Gradient updates during SFT push weights toward code-specific patterns. With 3 epochs, weights drift far from pre-trained values.
QLoRA Quantization Noise on Sensitive Layers
4-bit NF4 quantization introduces rounding errors that accumulate in first and last transformer layers. Precise token probabilities (choosing between = and ==) show measurable error rate increase.
NF4 maps fp16 values to nearest of 16 discrete values. In early layers, this error propagates through all subsequent layers.
vLLM Memory OOM at High Concurrency
At 32+ concurrent requests with long context (4K+ tokens), PagedAttention block allocator runs out of GPU memory. All in-flight requests affected simultaneously.
PagedAttention allocates blocks on demand. Under memory pressure, vLLM swaps to CPU RAM. At extreme concurrency, swap bandwidth becomes bottleneck.
Overconfident on Out-of-Domain Code
Trained primarily on Python/TS/Rust/SQL. For Kotlin, Haskell, or Solidity, generates syntactically plausible but subtly wrong code.
Fine-tuning increases confidence on in-domain distributions. Calibration for uncertainty on OOD inputs degrades.
Adapter Merge Changes Output Distribution
After merge_and_unload(), merged model produces slightly different outputs than pre-merge. Benchmark numbers may not match.
BitsAndBytes 4-bit dequantization introduces small errors during merge. Errors compound across generation process.
vLLM Output Non-Determinism
At temperature=0, different concurrency levels produce different outputs. Continuous batching changes which attention computations are fused together.
GPU floating-point operations are not strictly commutative. Fused attention kernels produce different rounding results depending on batch composition.