Async vector sync with status tracking #43

New Issue

2026-07-03T20:58:15+02:00

kasun commented

2026-07-03 20:58:15 +02:00

Context

POST /sync (SyncController → SyncService.syncServiceVectorStore) currently runs synchronously: the HTTP request blocks until every page of the Homebox catalog has been fetched, embedded, and written to pgvector. For now the manual "sync" button in the frontend relies on this (it just awaits the fetch and gates AI search on the promise resolving). This ticket tracks moving to the async version once we outgrow that.

Problem

Blocking the request thread for the full sync duration:

risks browser/proxy timeouts on larger inventories
ties up a request thread for the whole sync
gives the frontend no way to know sync progress if the user navigates away and comes back, or if we later add a periodic/scheduled sync that isn't triggered by a button at all

Goal

Make /sync non-blocking, and let the frontend know when an in-progress sync finishes so it can keep AI search disabled until indexing is actually done.

Scope

Backend

Run SyncService.syncServiceVectorStore asynchronously (@Async + dedicated TaskExecutor, not the request thread pool) so SyncController returns immediately.
Track sync status per connection: IDLE / SYNCING / COMPLETED / FAILED, plus lastSyncedAt. Start with an in-memory ConcurrentHashMap<connectionId, SyncStatus> — revisit persistence only if status needs to survive a backend restart.
SyncController returns 202 Accepted (not 204) and flips status to SYNCING before dispatching the async work.
Extend ConnectionStatusResponse (GET /connections/status) to include the new sync status fields — reuse this existing endpoint instead of adding a dedicated status endpoint, since the frontend already calls it.
On failure mid-sync (e.g. Homebox becomes unreachable), record FAILED instead of leaving status stuck at SYNCING.

Frontend

After triggering /sync, poll GET /connections/status at a short interval while status is SYNCING.
Keep AI search disabled until status is COMPLETED; surface FAILED with a retry action.

Out of scope

Periodic/scheduled sync (separate ticket — needs this status tracking as a dependency)
SSE/WebSocket push instead of polling

Acceptance criteria

Triggering sync from the UI returns immediately, no request blocking.
AI search stays disabled for a connection while its sync status is SYNCING.
/connections/status reflects live sync state on poll, without requiring a page refresh.
A failed sync surfaces as FAILED rather than hanging indefinitely.

### Context `POST /sync` (`SyncController` → `SyncService.syncServiceVectorStore`) currently runs synchronously: the HTTP request blocks until every page of the Homebox catalog has been fetched, embedded, and written to pgvector. For now the manual "sync" button in the frontend relies on this (it just awaits the fetch and gates AI search on the promise resolving). This ticket tracks moving to the async version once we outgrow that. ### Problem Blocking the request thread for the full sync duration: - risks browser/proxy timeouts on larger inventories - ties up a request thread for the whole sync - gives the frontend no way to know sync progress if the user navigates away and comes back, or if we later add a periodic/scheduled sync that isn't triggered by a button at all ### Goal Make `/sync` non-blocking, and let the frontend know when an in-progress sync finishes so it can keep AI search disabled until indexing is actually done. ### Scope **Backend** - Run `SyncService.syncServiceVectorStore` asynchronously (`@Async` + dedicated `TaskExecutor`, not the request thread pool) so `SyncController` returns immediately. - Track sync status per connection: `IDLE` / `SYNCING` / `COMPLETED` / `FAILED`, plus `lastSyncedAt`. Start with an in-memory `ConcurrentHashMap<connectionId, SyncStatus>` — revisit persistence only if status needs to survive a backend restart. - `SyncController` returns `202 Accepted` (not `204`) and flips status to `SYNCING` before dispatching the async work. - Extend `ConnectionStatusResponse` (`GET /connections/status`) to include the new sync status fields — reuse this existing endpoint instead of adding a dedicated status endpoint, since the frontend already calls it. - On failure mid-sync (e.g. Homebox becomes unreachable), record `FAILED` instead of leaving status stuck at `SYNCING`. **Frontend** - After triggering `/sync`, poll `GET /connections/status` at a short interval while status is `SYNCING`. - Keep AI search disabled until status is `COMPLETED`; surface `FAILED` with a retry action. ### Out of scope - Periodic/scheduled sync (separate ticket — needs this status tracking as a dependency) - SSE/WebSocket push instead of polling ### Acceptance criteria - Triggering sync from the UI returns immediately, no request blocking. - AI search stays disabled for a connection while its sync status is `SYNCING`. - `/connections/status` reflects live sync state on poll, without requiring a page refresh. - A failed sync surfaces as `FAILED` rather than hanging indefinitely.

kasun added the Kind/Feature

Priority

High

2

labels 2026-07-03 20:58:15 +02:00

kasun self-assigned this 2026-07-03 20:58:15 +02:00

kasun added this to the Advanced feature development project 2026-07-03 20:58:15 +02:00

Sign in to join this conversation.