CODESAGEZ/QLORA TRAINING PIPELINE

Adaptation & Training Dynamics

Full execution lifecycle for fine-tuning Llama-3.3-8B and Qwen2.5-Coder across 52K verified code pairs using QLoRA, DoRA, and NF4 quantization on Modal A100 GPUs ($9.40 total compute).

STAGE-01/

Dataset Curation

Quality over quantity

52K pairs · 4 sources · 3 filtering passes

sources
  • ›The Stack v2 (Python subset): Filtered: Pylint score < 7, syntax errors, <5 or >500 lines. Applied Google Python Style Guide refactoring.
  • ›CodeAlpaca-20k: Self-instruct generated. Filtered for correctness: removed pairs where code does not run.
  • ›Synthetic problem-solution pairs: GPT-4o generated 25K problems with unit tests. 82% pass rate. Most important for domain specialization.
filtering
  • ›Pass 1: Syntax check (ast.parse() for Python, tsc --noEmit for TS)
  • ›Pass 2: Execution check (sandbox, verify exit 0)
  • ›Pass 3: Deduplication (MinHash LSH at 80% Jaccard, removed ~4,200 duplicates)
distribution
train:46,800 (90%)
validation:2,600 (5%)
test:2,600 (5%)
STAGE-02/

QLoRA Configuration

4-bit NF4 · r=16 · $9.40 total cost

NF4 quantization · DoRA · all-linear targets

base Model

meta-llama/Llama-3.3-8B-Instruct (8.03B params, 32 layers, GQA)

quantization
type:NF4 (NormalFloat4)
compute:bfloat16
doubleQuant:
reason:NF4 is optimal for normally-distributed neural network weights.
lora
r:16
alpha:16
dropout:0.05
targets:all-linear (q/k/v/o_proj + gate/up/down_proj)
dora:
doraReason:+1-4.4% improvement over standard LoRA at same rank.
training
epochs:3
batchSize:2 (effective 8 with grad accum 4)
lr:2e-4 cosine decay
warmup:3% (~130 steps)
maxSeqLen:2048
packing:
optimizer:paged_adamw_8bit
hardware

1x NVIDIA A100 80GB · ~2.8 hours · $9.40 via Modal

STAGE-03/

W&B Experiment Tracking

3 runs · 12 hyperparameter trials

Loss curves, ablation results, production model selection

runs
  • ›codesage-v0.1: Baseline
  • ›codesage-v0.2: Improved
  • ›codesage-v1.0: Production
findings
  • ›all-linear targets vs attention-only: +2.3pp HumanEval
  • ›DoRA vs standard LoRA: +1.5pp HumanEval at same r
  • ›packing=True: 40% faster training, same final loss
  • ›r=32 vs r=16: +0.3pp HumanEval, 2.1x slower — not worth it
STAGE-04/

Adapter Merging + Hub Push

GGUF + ONNX + safetensors

Merge, quantize, publish to HuggingFace

steps
  • ›Load base model in fp16
  • ›Load QLoRA adapter
  • ›model.merge_and_unload() — merges W + (B×A) × (alpha/r)
  • ›Save in safetensors format (not pickle)
formats
  • ›safetensors (fp16):
  • ›GGUF q4_K_M:
  • ›ONNX + int8:
STAGE-05/

Modal Serverless GPU Deployment

Scales to 0 · No idle GPU cost

A10G GPU · container_idle_timeout=60 · ~95% cheaper than AWS

why Modal

AWS requires a minimum GPU instance running 24/7 = ~$2,400/month. Modal charges only when inference runs — ~95% cheaper for variable traffic.

config
gpu:A10G (24GB VRAM)
idleTimeout:60
concurrentInputs:32
coldStart:~18-25 seconds
warmLatency:<100ms overhead