Skip to content

08. Testing & Quality

Primary gate: parity

03_validate_parity.py compares PyTorch vs ONNX greedy decode on golden fixtures. Target: ≥99% token-exact. Documented runs hit 100% on 1100 fixtures per direction for fp32.

Secondary: precision benchmarks

07_benchmark_precision.py measures quantized models against fp32 ONNX. Results committed under fixtures/benchmark-*.json and summarized in BENCHMARKS.md / BENCHMARKS_1B.md.

Other signals

  • Smoke translation matrices (generate_translation_matrix.py)
  • Live browser benches via browser-lab
  • Visual reports (generate_visual_reports.py, generate_live_reports.py)

Absent

  • No pytest suite
  • No CI job running parity on PRs
  • README test_hf_models.py missing in this checkout

Interview Q&A

Q: How do you know Q4 is “good enough”?
A: Published token-match rates vs fp32 — 1B holds up better than 200M; choose precision by size/accuracy tradeoff tables.