Agentic AI Architecture

Runtime model

The agent is a client-side orchestration hook rather than a server process. useVoiceAgent owns lifecycle state and delegates inference to src/lib/llm-runtime.ts. A selected variant maps to one adapter: Gemma custom kernels, LFM custom kernels, Transformers.js, or WebLLM.

flowchart TD
  classDef default fill:#1e293b,stroke:#38bdf8,stroke-width:2px,color:#f8fafc
  classDef highlight fill:#065f46,stroke:#34d399,stroke-width:2px,color:#f0fdf4
  classDef warning fill:#78350f,stroke:#fbbf24,stroke-width:2px,color:#fffbeb
  Request["User request"]:::highlight --> Prompt["System prompt + recent history"]
  Prompt --> Adapter["Selected LLM adapter"]
  Adapter --> Events["text / thinking / tool events"]
  Events --> ToolGate{"Tools enabled?"}
  ToolGate -->|No| Answer["Answer stream"]
  ToolGate -->|Yes| Execute["Validate and execute bounded tools"]
  Execute --> Adapter
  Answer --> Split["Sentence splitter"]
  Split --> Speech["TTS queue"]
  Execute --> Answer
  linkStyle default stroke:#64748b,stroke-width:2px

Streaming and tools

streamLLMWithToolLoop caps tool rounds and calls per round. Models with native tool support receive structured turns; prompt-fallback models receive serialized tool-call and tool-result messages. A nudge prompt handles models that fail to emit a tool call when a request obviously needs time/date or calculation.

Events are normalized into text_delta, thinking_delta, tool_call, tool_result, and done. The UI can therefore render reasoning and tool activity without knowing which engine produced it.

Voice response path

When TTS is enabled, text deltas feed TextSplitterStream. The next sentence is prefetched while the current sentence synthesizes, PCM is enqueued for playback, and the final chunks are combined into a WAV object URL for the message bubble. Abort signals stop both model generation and playback.

State and failure handling

• Setup phases: selecting, loading, ready.
• Runtime statuses include listening, recording, transcribing, thinking, synthesizing, speaking, and error.
• A ref-backed abort controller prevents stale generations from updating the conversation.
• Barge-in aborts active generation when new speech is detected.
• Model switching unloads stale engine/model combinations.
• Tool execution is bounded and validated before dispatch.

There is no durable server checkpoint or cross-device memory. Conversation state is in React memory and is cleared explicitly or on page reload.

Model selection matrix

Engine	Adapter	Strength	Cost
Gemma kernel	gemma4-kernel	Custom WebGPU path and vision-capable variants	High memory and device sensitivity
LFM kernel	lfm2-kernel	Small custom-kernel models	Specialized implementation
Transformers.js	transformers-js	Hugging Face ONNX model family	Browser/WebGPU compatibility
WebLLM	webllm	MLC-compiled Qwen/Llama choices	Model-specific runtime footprint

The adapter boundary is not merely a type abstraction. load, unload, isReady, abort, stream, and getLoadProgress are the lifecycle operations needed to prevent a prior model from consuming memory or emitting events after a switch.

Event semantics

text_delta is user-visible answer content. thinking_delta is optional diagnostic/reasoning UI content. tool_call and tool_result are rendered as activity and also become the next model turn. done marks a generation boundary, while the hook separately waits for TTS completion before returning to ready/listening state.

Tool safety boundary

The built-in registry contains calculator and current-time capabilities. Tool execution is local and bounded by MAX_TOOL_ROUNDS and MAX_TOOL_CALLS_PER_ROUND. Prompt-fallback models receive serialized tool turns because not every small model supports structured tool calls. This compatibility path increases prompt-injection and parser risk, so experimental tools are opt-in.