05. Data Model & Storage

Persistence strategy

Confirmed: no SQL/ORM. Ephemeral Redis hashes for job status + local filesystem for inputs and artifacts (docs/decisions/architectural-decisions.md ADR 4; app/services/storage.py).

Domain entities

%%{init: {
  "theme": "base",
  "themeVariables": {
    "primaryColor": "#1e293b",
    "primaryTextColor": "#cad6eaff",
    "primaryBorderColor": "#38bdf8",
    "lineColor": "#64748b",
    "attributeBackgroundColor": "#1e293b",
    "attributeTextColor": "#f8fafc"
  },
  "themeCSS": ".er.attributeBoxOdd { fill: #1e293b !important; } .er.attributeBoxEven { fill: #0f172a !important; } .row-rect-odd { fill: #1e293b !important; } .row-rect-even { fill: #0f172a !important; } .row-rect-odd path { fill: #1e293b !important; } .row-rect-even path { fill: #0f172a !important; }"
}}%%
erDiagram
  JOB ||--o{ PAGE_METADATA : has
  PAGE_METADATA ||--o{ BLOCK_METADATA : contains
  JOB {
    string job_id PK
    string status
    int progress
    int total_blocks
    int current_page
    int total_pages
    string created_at
    string started_at
    string finished_at
    string error
    string result_files
  }
  PAGE_METADATA {
    int page
    string source_file
    string processed_at
    float processing_time_seconds
    int total_blocks
    int text_blocks
    int image_blocks
  }
  BLOCK_METADATA {
    int index
    string label
    float_list coordinate
    string content
    bool is_image_block
    float processing_time_seconds
  }

Pydantic models: JobStatusResponse, PageMetadata, BlockMetadata in app/schemas/ocr.py.

Job lifecycle states

stateDiagram-v2
  [*] --> pending: POST /ocr/jobs
  pending --> processing: Celery starts
  processing --> done: Artifacts written
  processing --> failed: Exception
  done --> [*]
  failed --> [*]

TTL: Redis key job:{id} expires after 86400 seconds (API create + task updates refresh expire).

Filesystem layout

jobs/{uuid4_hex}/
  input.<ext>
  result.md          # optional
  result.html        # optional
  result.zip         # optional (e.g. Sarvam)
  metadata/
    page_0001.json
    page_0002.json
    ...

• Job IDs: uuid.uuid4().hex, validated with UUID4 hex regex before mkdir (is_valid_job_id)
• List jobs: directories under JOBS_DIR sorted by mtime; status fields merged from Redis (missing Redis → treat as pending)

Redis vs Celery DB split

Logical DB	Use
From REDIS_URL (typically /0)	Job status hashes
Rewritten /1	Celery broker + result backend

Evidence: app/tasks/celery_app.py _make_celery_redis_url.

Artifact / download flow

graph LR
  classDef default fill:#1e293b,stroke:#38bdf8,stroke-width:2px,color:#f8fafc
  classDef highlight fill:#065f46,stroke:#34d399,stroke-width:2px,color:#f0fdf4

  U[Upload] --> I[input.ext]
  W[Worker] --> M[metadata/page_*.json]
  W --> MD[result.md]
  W --> HTML[result.html]
  D["GET download/zip"]:::highlight --> Z[Temp ZIP]
  Z --> MD
  Z --> HTML
  Z --> M

  linkStyle default stroke:#64748b,stroke-width:2px

ZIP is built on demand from md + html + metadata (_build_job_bundle_zip in app/api/ocr.py), served via FileResponse with background temp cleanup. Download requires Redis status=done (409 otherwise).

Consistency and idempotency

• Job IDs are unique UUIDs — submit is not idempotent by content hash
• Delete removes disk tree + Redis key
• Inferred gap: after Redis TTL expiry, directories may remain; list may show jobs with weak/default status
• No transactional link between Redis and disk beyond best-effort task updates

Data-model interview Q&A

Q: Why no Postgres?
A: Service is an on-demand OCR utility, not a document store. Redis + disk keeps infra minimal (ADR 4).

Q: How is progress exposed?
A: Worker increments Redis hash fields (progress, current_page, total_blocks); API maps them to JobStatusResponse.

Q: How do you prevent path traversal?
A: Job IDs must match UUID4 hex; paths are always jobs_dir / job_id / ....