Unique Engineering Highlights¶

1. One streaming contract over multiple browser LLMs¶

src/lib/llm-runtime.ts maps custom kernels, Transformers.js, and WebLLM to one adapter interface. This isolates provider-specific load, abort, unload, and stream behavior while keeping the agent loop stable.

2. Sentence-level streaming speech¶

TextSplitterStream feeds TTS before the entire response exists. Prefetching the next sentence overlaps synthesis with playback and reduces perceived latency, at the cost of more cancellation and partial-audio cleanup.

3. Voice barge-in¶

The STT worker’s recording status can abort generation and stop TTS. This makes the interaction conversational rather than request/response, but requires ref-backed lifecycle state to avoid races.

4. Browser-native execution isolation¶

AudioWorklet and Web Worker boundaries keep microphone processing and transcription work away from React rendering. COOP/COEP headers enable the threaded WASM path.

5. Bounded local tool loop¶

The runtime supports structured tools and prompt-fallback tools, validates calls, limits rounds, and nudges models that fail to emit the expected format. This is a pragmatic compatibility layer for heterogeneous small models.

6. Memory-aware model lifecycle¶

The app loads models only after setup, tracks progress, unloads stale variants, disposes engines on reset/unmount, and revokes generated audio URLs. These are essential in a browser where memory is shared with the tab.

7. Device-sensitive defaults and presets¶

The setup layer chooses a smaller iOS default and offers Fast & Light, Balanced, and Flagship presets. This turns hardware variability into an explicit product choice instead of allowing every user to discover memory limits by failure.

8. Multilingual voice alignment¶

Supertonic language selection, Hindi/Hinglish detection, Roman-to-Devanagari typing, and gender-aware prompt phrasing form a coherent language experience. The value is not a single translation feature; it is alignment between input, model response, and synthesized voice.

9. User-visible inference telemetry¶

Message metrics include time to first token, tokens per second, and total tokens. These measurements make local inference behavior explainable to users and useful for comparing model presets, while still avoiding a centralized telemetry service.