05. Data Model and Storage¶

Core entities¶

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#1e293b",
    "primaryTextColor": "#cad6eaff",
    "primaryBorderColor": "#38bdf8",
    "lineColor": "#64748b",
    "attributeBackgroundColor": "#1e293b",
    "attributeTextColor": "#f8fafc"
  },
  "themeCSS": ".er.attributeBoxOdd { fill: #1e293b !important; } .er.attributeBoxEven { fill: #0f172a !important; } .row-rect-odd { fill: #1e293b !important; } .row-rect-even { fill: #0f172a !important; } .row-rect-odd path { fill: #1e293b !important; } .row-rect-even path { fill: #0f172a !important; }"
}}%%
erDiagram
  PROJECTS ||--o{ DOCUMENTS : contains
  PROJECTS ||--o{ QUERY_HISTORY : records
  DOCUMENTS ||--o{ CHUNKS : split_into
  COLLECTIONS ||--o{ DOCUMENTS : optional

  PROJECTS {
    text id PK
    text name
    text embedding_model_id
    int chunk_size
    int chunk_overlap
    int retrieval_top_k
    boolean hybrid_retrieval_enabled
  }
  DOCUMENTS {
    text id PK
    text project_id FK
    text collection_id FK
    text status
    text error_message
    text metadata_json
  }
  CHUNKS {
    text id PK
    text document_id FK
    int chunk_index
    text text
    vector embedding
    text embedding_model_id
    text metadata_json
  }
  QUERY_HISTORY {
    text id PK
    text project_id FK
    text query
    text answer
    text retrieved_chunks_json
  }
  SETTINGS {
    text key PK
    text value
  }

Schema source: src/db/migrations.ts (version 1).

Tables in practice¶

Table	Used by app?	Notes
`projects`	Yes	Embedding model + retrieval settings
`documents`	Yes	Status machine for indexing
`chunks`	Yes	Text + vector + metadata
`query_history`	Yes	Chat persistence
`settings`	Yes	Export sync of UI prefs
`collections`	No writes found	Schema only
`index_jobs`	No writes found	Schema only
`model_cache`	No usage found	Schema only
`migration_versions`	Yes	Migration bookkeeping

Indexes¶

idx_documents_project, idx_chunks_document, idx_chunks_doc_index
GIN: to_tsvector('english', text) on chunks
No HNSW/IVFFlat on embedding — similarity uses <=> ordered scans

Persistence layout¶

Store	Mechanism	Contents
PGlite	`idb://browser-rag`	SQL tables + vectors
File IDB	`browser-rag-files` / store `files`	Original upload bytes for retry
Preferences	`localStorage` key `browser-rag-preferences`	Active project, LLM ids

Artifact / file lifecycle¶

graph TD
  classDef default fill:#1e293b,stroke:#38bdf8,stroke-width:2px,color:#f8fafc
  classDef highlight fill:#065f46,stroke:#34d399,stroke-width:2px,color:#f0fdf4
  classDef warning fill:#78350f,stroke:#fbbf24,stroke-width:2px,color:#fffbeb

  Upload["User upload"]:::highlight
  DocRow["documents row pending"]
  FileIDB["saveDocumentFile IDB"]
  Worker["indexing.worker extract+chunk"]
  Embed["embedding.worker"]
  Chunks["chunks rows + vectors"]:::highlight
  Fail["status failed"]:::warning
  Retry["retry from IDB bytes"]
  Export["exportDb tar.gz"]
  Gap["File IDB not in export"]:::warning

  Upload --> DocRow
  Upload --> FileIDB
  DocRow --> Worker --> Embed --> Chunks
  Embed --> Fail
  Fail --> Retry --> Worker
  Chunks --> Export
  FileIDB -.-> Gap

  linkStyle default stroke:#64748b,stroke-width:2px

Export / import¶

exportDb: sync prefs → settings, then dumpDataDir() → .tar.gz
importDb: close DB, delete IndexedDB names containing browser-rag, loadDataDir, run migrations, restore prefs to localStorage
Gap: browser-rag-files is a separate database and is not part of the PGlite dump — retries after restore may require re-upload

Consistency and idempotency¶

Re-index deletes existing chunks for the document inside a transaction before insert
Retrieval always filters by embedding model id to avoid mixing dimensions
Query history append-only; no update-in-place for answers mid-stream

Data-model interview questions¶

Q: How do you store vectors in the browser?
A: PGlite with the pgvector extension; embeddings inserted as vector literals like [...].

Q: Why keep original files separately?
A: Failed indexes can retry without asking the user to re-select the file (document-files.ts).

Q: What happens at scale?
A: Without an ANN index, each query scans candidate vectors with <=>. Fine for personal corpora; risky for very large chunk counts.