CODESAGEZ/QLORA TRAINING PIPELINE
Adaptation & Training Dynamics
Full execution lifecycle for fine-tuning Llama-3.3-8B and Qwen2.5-Coder across 52K verified code pairs using QLoRA, DoRA, and NF4 quantization on Modal A100 GPUs ($9.40 total compute).
STAGE-01/
Dataset Curation
Quality over quantity
52K pairs · 4 sources · 3 filtering passes
sources
- ›The Stack v2 (Python subset): Filtered: Pylint score < 7, syntax errors, <5 or >500 lines. Applied Google Python Style Guide refactoring.
- ›CodeAlpaca-20k: Self-instruct generated. Filtered for correctness: removed pairs where code does not run.
- ›Synthetic problem-solution pairs: GPT-4o generated 25K problems with unit tests. 82% pass rate. Most important for domain specialization.
filtering
- ›Pass 1: Syntax check (ast.parse() for Python, tsc --noEmit for TS)
- ›Pass 2: Execution check (sandbox, verify exit 0)
- ›Pass 3: Deduplication (MinHash LSH at 80% Jaccard, removed ~4,200 duplicates)
distribution
train:46,800 (90%)
validation:2,600 (5%)
test:2,600 (5%)
STAGE-02/
QLoRA Configuration
4-bit NF4 · r=16 · $9.40 total cost
NF4 quantization · DoRA · all-linear targets
base Model
meta-llama/Llama-3.3-8B-Instruct (8.03B params, 32 layers, GQA)
quantization
type:NF4 (NormalFloat4)
compute:bfloat16
doubleQuant:
reason:NF4 is optimal for normally-distributed neural network weights.
lora
r:16
alpha:16
dropout:0.05
targets:all-linear (q/k/v/o_proj + gate/up/down_proj)
dora:
doraReason:+1-4.4% improvement over standard LoRA at same rank.
training
epochs:3
batchSize:2 (effective 8 with grad accum 4)
lr:2e-4 cosine decay
warmup:3% (~130 steps)
maxSeqLen:2048
packing:
optimizer:paged_adamw_8bit
hardware
1x NVIDIA A100 80GB · ~2.8 hours · $9.40 via Modal
STAGE-03/
W&B Experiment Tracking
3 runs · 12 hyperparameter trials
Loss curves, ablation results, production model selection
runs
- ›codesage-v0.1: Baseline
- ›codesage-v0.2: Improved
- ›codesage-v1.0: Production
findings
- ›all-linear targets vs attention-only: +2.3pp HumanEval
- ›DoRA vs standard LoRA: +1.5pp HumanEval at same r
- ›packing=True: 40% faster training, same final loss
- ›r=32 vs r=16: +0.3pp HumanEval, 2.1x slower — not worth it
STAGE-04/
Adapter Merging + Hub Push
GGUF + ONNX + safetensors
Merge, quantize, publish to HuggingFace
steps
- ›Load base model in fp16
- ›Load QLoRA adapter
- ›model.merge_and_unload() — merges W + (B×A) × (alpha/r)
- ›Save in safetensors format (not pickle)
formats
- ›safetensors (fp16):
- ›GGUF q4_K_M:
- ›ONNX + int8:
STAGE-05/
Modal Serverless GPU Deployment
Scales to 0 · No idle GPU cost
A10G GPU · container_idle_timeout=60 · ~95% cheaper than AWS
why Modal
AWS requires a minimum GPU instance running 24/7 = ~$2,400/month. Modal charges only when inference runs — ~95% cheaper for variable traffic.
config
gpu:A10G (24GB VRAM)
idleTimeout:60
concurrentInputs:32
coldStart:~18-25 seconds
warmLatency:<100ms overhead