Backend Deep Dive¶
Backend boundary¶
There is no Python backend, API router, database service, queue, or server-side authentication in this repository. The “backend” responsibilities are browser runtime services: workers, model adapters, tool functions, and storage helpers.
Entry-point and module map¶
| Area | Files | Responsibility |
|---|---|---|
| Shell | src/main.tsx, src/App.tsx |
Mounting, tabs, responsive layout |
| Orchestration | src/hooks/use-voice-agent.ts |
Session lifecycle and request coordination |
| LLM | src/lib/llm-runtime.ts, src/lib/llm/ |
Provider adapters, parsers, streaming |
| STT | public/stt-worker-esm.js, src/lib/stt-models.ts |
Worker model loading and transcription |
| Audio | public/vad-processor.js, src/lib/piper/ |
Frames, WAV conversion, playback |
| TTS | src/hooks/use-tts.ts, src/lib/tts-providers/ |
Engine loading and synthesis |
| Tools | src/lib/tools/ |
Registry, validation, execution |
| Preferences | src/lib/user-preferences.ts |
Local session blueprint |
Request and cancellation lifecycle¶
The hook creates a generation abort controller before requesting an answer. It appends a placeholder assistant message, forwards normalized stream events into state, and clears the controller in finally. stopGeneration aborts the model and stops TTS; endCall additionally stops the microphone graph and worker. Reset/unmount unloads every engine and terminates the worker.
This explicit cleanup is the browser equivalent of request cancellation plus resource disposal. Without it, a model switch could leave old WebGPU allocations or an old worker delivering transcripts to a new session.
Error boundaries¶
Model-load errors become status messages with targeted advice, including memory guidance for Gemma. Microphone errors are surfaced as status errors. Worker errors stop transcription and set a visible error. TTS errors are logged and reflected in the status message. There is no remote telemetry or centralized log aggregation.
Runtime boundaries¶
| Boundary | Responsibility |
|---|---|
| React hook | Coordinates user intent and lifecycle |
| Audio worklet | Converts microphone input into audio frames |
| STT worker | Runs VAD, model loading, and transcription |
| LLM adapter | Normalizes model load, stream, abort, and unload |
| Tool registry | Validates and runs local capabilities |
| TTS provider | Loads voice assets and returns PCM |
The explicit LLMBackendHandle interface makes model providers replaceable. It exposes readiness, progress, load, unload, abort, and streaming methods, which are the key lifecycle operations for large in-browser models.
Configuration and errors¶
Vite aliases @ to src, sets an ESNext build target, and excludes model runtimes from dependency pre-bundling. Errors are surfaced through hook state and status banners; lower-level failures also log to the browser console.
Interview questions¶
Why no API layer? Privacy and offline operation are product requirements. The cost is large downloads, browser capability constraints, and no centralized observability.
How would you add a remote provider? Implement the same adapter contract or replace one adapter’s stream method with an authenticated fetch while preserving normalized events and cancellation.
What is the concurrency model? React handles UI state; STT runs in a worker; audio processing runs in an AudioWorklet; model and TTS work are asynchronous and guarded by abort/unload lifecycle code.