Phase 1: Knowledge Acquisition

Training & Fine-Tuning

A comprehensive look at the end-to-end pipeline for turning a base Llama 3.3 8B model into a highly-specialized code completion engine.

Dataset Curation

Quality over quantity

52K pairs · 4 sources · 3 filtering passes

sources

  • The Stack v2 (Python subset) - Filtered: Pylint score < 7, syntax errors, <5 or >500 lines. Applied Google Python Style Guide refactoring.
  • CodeAlpaca-20k - Self-instruct generated. Filtered for correctness: removed pairs where code does not run.
  • Synthetic problem-solution pairs - GPT-4o generated 25K problems with unit tests. 82% pass rate. Most important for domain specialization.

filtering

  • Pass 1: Syntax check (ast.parse() for Python, tsc --noEmit for TS)
  • Pass 2: Execution check (sandbox, verify exit 0)
  • Pass 3: Deduplication (MinHash LSH at 80% Jaccard, removed ~4,200 duplicates)

distribution

train:46,800 (90%)
validation:2,600 (5%)
test:2,600 (5%)

QLoRA Configuration

4-bit NF4 · r=16 · $9.40 total cost

NF4 quantization · DoRA · all-linear targets

base Model

meta-llama/Llama-3.3-8B-Instruct (8.03B params, 32 layers, GQA)

quantization

type:NF4 (NormalFloat4)
compute:bfloat16
doubleQuant:
reason:NF4 is optimal for normally-distributed neural network weights.

lora

r:16
alpha:16
dropout:0.05
targets:all-linear (q/k/v/o_proj + gate/up/down_proj)
dora:
doraReason:+1-4.4% improvement over standard LoRA at same rank.

training

epochs:3
batchSize:2 (effective 8 with grad accum 4)
lr:2e-4 cosine decay
warmup:3% (~130 steps)
maxSeqLen:2048
packing:
optimizer:paged_adamw_8bit

hardware

1x NVIDIA A100 80GB · ~2.8 hours · $9.40 via Modal

W&B Experiment Tracking

3 runs · 12 hyperparameter trials

Loss curves, ablation results, production model selection

runs

  • codesage-v0.1 - Baseline
  • codesage-v0.2 - Improved
  • codesage-v1.0 - Production

findings

  • all-linear targets vs attention-only: +2.3pp HumanEval
  • DoRA vs standard LoRA: +1.5pp HumanEval at same r
  • packing=True: 40% faster training, same final loss
  • r=32 vs r=16: +0.3pp HumanEval, 2.1x slower — not worth it

Adapter Merging + Hub Push

GGUF + ONNX + safetensors

Merge, quantize, publish to HuggingFace

steps

  • Load base model in fp16
  • Load QLoRA adapter
  • model.merge_and_unload() — merges W + (B×A) × (alpha/r)
  • Save in safetensors format (not pickle)

formats

  • safetensors (fp16) -
  • GGUF q4_K_M -
  • ONNX + int8 -

Modal Serverless GPU Deployment

Scales to 0 · No idle GPU cost

A10G GPU · container_idle_timeout=60 · ~95% cheaper than AWS

why Modal

AWS requires a minimum GPU instance running 24/7 = ~$2,400/month. Modal charges only when inference runs — ~95% cheaper for variable traffic.

config

gpu:A10G (24GB VRAM)
idleTimeout:60
concurrentInputs:32
coldStart:~18-25 seconds
warmLatency:<100ms overhead