Skip to content

03. Backend Deep Dive

There is no FastAPI service. “Backend” means the Python experiment pipeline under scripts/.

Entrypoints

Script Role
config.py Load config.yaml singleton
sample_sft_prompts.py Stratified prompts + leakage blocklist
generate_traces.py MLX raw CoT; resume by id
compress_traces.py Async OpenAI-compatible compression
validate_traces.py Style + logic gates
grug_score.py Quantitative Grug metrics
format_data.py SFT JSONL + regularization
train.py Wrap python -m mlx_lm lora
eval.py GSM8K baseline / fine-tuned
generate.py Ad-hoc generation smoke test
plot_results.py Charts into report/
sync_hf.py Stage/push HF repo
prompt_utils.py / generation_utils.py Shared helpers

Config

config.yaml: target model name/path, sampling sizes (StrategyQA, LogiQA, BoolQ, ANLI, PIQA, ReClor), seeds, temperatures, path templates.

Current checkout: deepseek-r1-7b / DeepSeek-R1-Distill-Qwen-7B-4bit.

Published runs: 1.5B paths in README/reports/HF staging.

lora_config.yaml: rank 16, alpha 32, scale 2.0, dropout 0.05; q/k/v/o proj; batch 2, grad_accum 2, iters 1000, lr 5e-6, grad checkpointing.

Training wrapper behavior

train.py builds the mlx_lm command, streams logs, tracks train/val loss, promotes best validation checkpoint to best_adapters.safetensors / adapters.safetensors, writes metrics.json and loss plot under the run directory.

Compression and validation

  • System prompt content comes from style_guide.md
  • Env: OPENAI_API_KEY, OPENAI_API_BASE, OPENAI_MODEL
  • grug_score.py combines compression ratio, article density, meta phrases, fragment length, repetition
  • validate_traces.py rejects low scores and logic failures (numeric facts, MC letters, incomplete tails)

SFT formatting tricks

  • Apply chat template then manually append <think>... so the template does not strip thinking
  • Regularization defaults: ~30% negative (raw thinking), ~20% system-prompt dropout on positives, half of negatives keep system prompt

Eval protocol

  • Dataset: GSM8K test (eval.py)
  • Optional style system prompt (default on; --no-system-prompt available)
  • Metrics: accuracy, thinking tokens, latency, format compliance
  • Makefile targets: eval-base, eval-ft, eval-all, plot

Interview Q&A

Q: Why wrap mlx_lm instead of a custom trainer?
A: Reuse a maintained MLX LoRA path; invest engineering in data quality and eval.

Q: Why compress with another LLM?
A: Cheaper than hand-writing thousands of Grug traces; style guide keeps outputs consistent enough to validate automatically.