03. Backend Deep Dive

No HTTP API. Pipeline scripts under src/ orchestrated by Makefile.

Numbered stages

Script	Role
01_export_encoder_decoder.py	Manual ONNX export (distilled)
v2/01_export_encoder_decoder.py	1B-aware export
onnx_bundle_optimize.py	Dedup, fuse, externalize, share decoder data
02_build_fast_tokenizers.py	SPM → JSON + dict ID remap
03_validate_parity.py / v2/03_*	PyTorch vs ONNX ≥99%
04_quantize_int8.py	Dynamic INT8
05_convert_fp16.py / v2/05_*	FP16 conversion
06_quantize_q4f16.py / v2/06_*	MatMulNBits Q4F16
05_upload_hf.py / v2/05_upload_hf.py	Model card + hf upload
07_benchmark_precision.py	Quant vs fp32 ONNX oracle
it2_onnx_wrappers.py	Encoder/decoder/past wrappers
it2_inference.py	Batched greedy decode for parity
translate.py	Shippable Python IndicTransONNX helper

Export design

• Optimum rejected → torch.onnx.export with dynamo=False
• Wrappers preserve encoder_attention_mask via zero-cost dependency trick
• decoder_with_past uses dummy encoder hidden states; real cross-attn KV from past
• Distilled path hardcodes head dims; v2 reads config for 1B

Optimize layout

Produces encoder_model.onnx(.data), decoder_model.onnx, decoder_with_past_model.onnx, decoder_shared.onnx.data, plus tokenizer/meta JSON. Documented ~40% size reduction on fp32 distilled bundles.

Make knobs

HF_ORG, SCRATCH, ONNX_OPSET=17, EVAL_BATCH_SIZE, Q4F16_BLOCK_SIZE, upload commit message.

Interview Q&A

Q: Why three ONNX files?
A: Matches encoder–decoder generation: encode once, first decode step, then past-key-value steps — what ORT Web demos consume.