Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8184db0c46 | ||
|
|
65ff109800 | ||
|
|
48d120fbff |
@@ -49,13 +49,15 @@ Package-by-feature layout. Server context path is `/api`. Main endpoints:
|
|||||||
- `GET /api/connections/status` — lists connected services for the current session
|
- `GET /api/connections/status` — lists connected services for the current session
|
||||||
- `DELETE /api/connections/{serviceType}` — removes a service from the session; invalidates the session if no connections remain
|
- `DELETE /api/connections/{serviceType}` — removes a service from the session; invalidates the session if no connections remain
|
||||||
- `POST /api/search` — paged search against the requested service; returns 401 if no active session
|
- `POST /api/search` — paged search against the requested service; returns 401 if no active session
|
||||||
|
- `POST /api/sync` — triggers a vector-store sync for the requested service; returns 401 if no active session. Not yet triggered on login or called from the frontend — manual/internal trigger only for now.
|
||||||
|
|
||||||
Six packages:
|
Eight packages:
|
||||||
|
|
||||||
**`shared/`** — cross-cutting types used by more than one feature package
|
**`shared/`** — cross-cutting types used by more than one feature package
|
||||||
|
|
||||||
- `ServiceType` (enum): identifies each integrated app (e.g. `HOMEBOX`); used in both `connection/` and `search/`
|
- `ServiceType` (enum): identifies each integrated app (e.g. `HOMEBOX`); used across `connection/`, `search/`, `sync/`
|
||||||
- `ServiceProvider` (interface): base for `ConnectionProvider` and `SearchProvider`; declares `getServiceType()`
|
- `ServiceProvider` (interface): base for `ConnectionProvider`, `SearchProvider`, `SyncProvider`; declares `getServiceType()`
|
||||||
|
- `ServiceItem`: normalized item shape (`id`, `title`, `description`, `extraData`) returned by both search and sync fetches
|
||||||
- `Endpoint` (enum): API path constants for all external service calls
|
- `Endpoint` (enum): API path constants for all external service calls
|
||||||
- `SessionKeys`: builds session attribute names of the form `{SERVICE_TYPE}_CONNECTION_ID`
|
- `SessionKeys`: builds session attribute names of the form `{SERVICE_TYPE}_CONNECTION_ID`
|
||||||
|
|
||||||
@@ -67,29 +69,40 @@ Six packages:
|
|||||||
**`connection/`** — connecting to and persisting service credentials
|
**`connection/`** — connecting to and persisting service credentials
|
||||||
|
|
||||||
- `ConnectionProvider` interface: extends `ServiceProvider`; each integrated app implements `login()` and credential checking
|
- `ConnectionProvider` interface: extends `ServiceProvider`; each integrated app implements `login()` and credential checking
|
||||||
|
- `ConnectionIdentifiable` interface: `appUrl()`/`username()`/`serviceType()`; implemented by `SearchRequest` and `SyncRequest` so `HomeboxItemClient` can resolve the underlying connection regardless of which feature is calling it
|
||||||
- `ConnectionService`: auto-discovers providers via Spring injection, dispatches login by `ServiceType`
|
- `ConnectionService`: auto-discovers providers via Spring injection, dispatches login by `ServiceType`
|
||||||
- `ConnectionController`: stores `{serviceType}_CONNECTION_ID` in `HttpSession` after login; reads session attributes to build status responses
|
- `ConnectionController`: stores `{serviceType}_CONNECTION_ID` in `HttpSession` after login; reads session attributes to build status responses
|
||||||
- Entity (`ConnectionEntity`) uses **Single Table Inheritance** — one `connections` table with app-specific nullable columns
|
- Entity (`ConnectionEntity`) uses **Single Table Inheritance** — one `connections` table with app-specific nullable columns
|
||||||
- `HomeboxConnectionProvider` / `HomeboxEntity`: Homebox-specific implementation
|
- `HomeboxConnectionProvider` / `HomeboxConnectionEntity`: Homebox-specific implementation
|
||||||
|
|
||||||
**`ai/`** — shared AI infrastructure used by multiple features (search, future image analysis)
|
**`homebox/`** — shared Homebox API access
|
||||||
|
|
||||||
- `EmbeddingService`: wraps Spring AI's `EmbeddingModel`; reused by any feature that needs to generate vectors
|
- `HomeboxItemClient`: fetches a page of items from the Homebox entities API and maps them to `ServiceItem`; used by both `HomeboxSearchProvider` (keyword search) and `HomeboxSyncProvider` (vector-store sync) so the fetch/mapping logic isn't duplicated
|
||||||
- `VectorStoreConfig`: Spring bean configuration for `PgVectorStore` (pgvector-backed `VectorStore`)
|
|
||||||
- All classes here are provider-agnostic — the OpenAI starter is pointed at LiteLLM, so the underlying model is configurable without code changes
|
**`vector/`** — vectorization
|
||||||
|
|
||||||
|
- `EmbeddingService`: embeds `ServiceItem`s into pgvector's `VectorStore` and prunes entries that no longer exist upstream. Document ID is `{connectionId}:{itemId}`, so re-syncing upserts existing items instead of duplicating them. Deliberately stateless (only field is the injected `VectorStore`) since it's a singleton bean — the per-sync ID accumulator used for pruning is owned by the caller (`HomeboxSyncProvider`), not held as instance state
|
||||||
|
- `PgVectorStore` itself is auto-configured by `spring-ai-starter-vector-store-pgvector` from `application.yaml` (`spring.ai.vectorstore.pgvector.dimensions`) — no manual config class in this codebase
|
||||||
|
|
||||||
**`search/`** — querying connected services (keyword and AI)
|
**`search/`** — querying connected services (keyword and AI)
|
||||||
|
|
||||||
- `SearchProvider` interface: extends `ServiceProvider`; each integrated app implements `getSearchResults()`
|
- `SearchProvider` interface: extends `ServiceProvider`; each integrated app implements `getSearchResults()`
|
||||||
- `SearchService`: maintains two provider maps — keyword providers and AI providers; routes based on `SearchRequest.aiSearch` flag
|
- `SearchService`: dispatches by `ServiceType` via a single provider map — **does not yet route on `SearchRequest.aiSearch`**; `HomeboxSearchProvider` and `HomeboxAiSearchProvider` both currently register for `ServiceType.HOMEBOX`, so only one wins the registration (known gap, pending AI search work)
|
||||||
- `SearchController`: guards with session check before delegating to `SearchService`
|
- `SearchController`: guards with session check before delegating to `SearchService`
|
||||||
- `SearchRequest`: includes `aiSearch: boolean` — when true, routes to AI provider instead of keyword provider
|
- `SearchRequest`: includes `aiSearch: boolean` (not yet consumed, see above) and implements `ConnectionIdentifiable`
|
||||||
- `PagedSearchResponse`: includes nullable `summary` field — populated only for AI search results; null for keyword search
|
- `PagedSearchResponse`: includes nullable `summary` field — populated only for AI search results; null for keyword search
|
||||||
- `HomeboxSearchProvider`: keyword search via Homebox API; unchanged from original implementation
|
- `HomeboxSearchProvider`: keyword search; delegates the remote fetch to `HomeboxItemClient`
|
||||||
- `HomeboxAiSearchProvider`: AI search via pgvector similarity; returns ranked items + generated summary
|
- `HomeboxAiSearchProvider`: **stub only** — `getSearchResults` throws `UnsupportedOperationException`; the actual similarity search + summary generation is pending
|
||||||
- `HomeboxSyncService`: fetches all Homebox items page by page, embeds them via `EmbeddingService`, stores in `VectorStore`; triggered on connection login (background sync)
|
|
||||||
|
|
||||||
**AI search flow:** on login → background sync indexes all Homebox items into pgvector. On AI search → embed query → pgvector similarity search → top N results passed to `ChatClient` for summary generation → return list + summary. Sync is idempotent (delete-then-reindex per connection).
|
**`sync/`** — indexing connected services into the vector store
|
||||||
|
|
||||||
|
- `SyncProvider` interface: extends `ServiceProvider`; each integrated app implements `syncVectorStore()`
|
||||||
|
- `SyncService`: dispatches by `ServiceType` via a provider map, same pattern as `SearchService`
|
||||||
|
- `SyncController`: session-gated `POST /sync`; not currently called by the frontend or triggered on login
|
||||||
|
- `SyncRequest`: implements `ConnectionIdentifiable`
|
||||||
|
- `HomeboxSyncProvider`: pages through the full Homebox catalog via `HomeboxItemClient`, embedding each page through `EmbeddingService.vectorizeData` and collecting every item ID seen along the way. Once the full page loop finishes, calls `EmbeddingService.deleteStaleVectorEntries` **once** with the complete ID set, removing any previously indexed item no longer present upstream. Order matters — pruning per page instead of once at the end would treat items on other pages as stale and delete them too.
|
||||||
|
|
||||||
|
**Pending work (not yet implemented):** `HomeboxAiSearchProvider`'s actual similarity search + summary generation; `SearchService` routing two provider maps by `aiSearch`; triggering `/sync` on login or from the frontend; a frontend sync button and AI search UI.
|
||||||
|
|
||||||
**`exception/`** — `GlobalExceptionHandler` via `@ControllerAdvice`
|
**`exception/`** — `GlobalExceptionHandler` via `@ControllerAdvice`
|
||||||
|
|
||||||
@@ -110,7 +123,7 @@ React 19 + TypeScript + SCSS, Vite 6 build. Package-by-feature under `components
|
|||||||
|
|
||||||
- PostgreSQL + pgvector (semantic search via embeddings); also used as the Spring Session store (JDBC)
|
- PostgreSQL + pgvector (semantic search via embeddings); also used as the Spring Session store (JDBC)
|
||||||
- LiteLLM as a unified AI proxy; Spring AI OpenAI starter wired to it — `OPENAI_BASE_URL` points to LiteLLM, not OpenAI directly, keeping the underlying model provider configurable
|
- LiteLLM as a unified AI proxy; Spring AI OpenAI starter wired to it — `OPENAI_BASE_URL` points to LiteLLM, not OpenAI directly, keeping the underlying model provider configurable
|
||||||
- `spring-ai-starter-vector-store-pgvector` provides `PgVectorStore`; configured in `ai/VectorStoreConfig`
|
- `spring-ai-starter-vector-store-pgvector` provides `PgVectorStore`, auto-configured from `application.yaml` (no manual config class)
|
||||||
- Embedding dimensions must stay consistent with the configured LiteLLM embedding model — changing models requires re-syncing all indexed items
|
- Embedding dimensions must stay consistent with the configured LiteLLM embedding model — changing models requires re-syncing all indexed items
|
||||||
- Processing pipeline (Phase 2): stage in DB → LLM inference → refine via UI → export to target app
|
- Processing pipeline (Phase 2): stage in DB → LLM inference → refine via UI → export to target app
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,17 @@
|
|||||||
|
package com.vaessl.app.connection;
|
||||||
|
|
||||||
|
import com.vaessl.app.shared.ServiceType;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Identifies the connection a request targets. Implemented by request records
|
||||||
|
* (e.g. {@code SearchRequest}, {@code SyncRequest}) so a single {@code HomeboxItemClient} can
|
||||||
|
* resolve the underlying connection regardless of which feature is calling it.
|
||||||
|
*/
|
||||||
|
public interface ConnectionIdentifiable {
|
||||||
|
|
||||||
|
String appUrl();
|
||||||
|
|
||||||
|
String username();
|
||||||
|
|
||||||
|
ServiceType serviceType();
|
||||||
|
}
|
||||||
@@ -0,0 +1,105 @@
|
|||||||
|
package com.vaessl.app.homebox;
|
||||||
|
|
||||||
|
import java.util.HashMap;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import org.springframework.data.domain.Page;
|
||||||
|
import org.springframework.data.domain.PageImpl;
|
||||||
|
import org.springframework.data.domain.Pageable;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
import org.springframework.web.client.RestClient;
|
||||||
|
import com.vaessl.app.connection.ConnectionEntity;
|
||||||
|
import com.vaessl.app.connection.ConnectionIdentifiable;
|
||||||
|
import com.vaessl.app.connection.ConnectionRepository;
|
||||||
|
import com.vaessl.app.connection.HomeboxConnectionEntity;
|
||||||
|
import com.vaessl.app.exception.ConnectionNotFoundException;
|
||||||
|
import com.vaessl.app.exception.RemoteApiException;
|
||||||
|
import com.vaessl.app.shared.ServiceItem;
|
||||||
|
import static com.vaessl.app.shared.Endpoint.HOMEBOX_QUERY_ALL_ITEMS;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fetches items from the Homebox entities API. Shared by {@code HomeboxSearchProvider} and
|
||||||
|
* {@code HomeboxSyncProvider} so both keyword search and vector-store sync page through the same
|
||||||
|
* remote call and item mapping.
|
||||||
|
*/
|
||||||
|
@Component
|
||||||
|
public class HomeboxItemClient {
|
||||||
|
|
||||||
|
private final RestClient.Builder restClientBuilder;
|
||||||
|
|
||||||
|
private final ConnectionRepository cRepository;
|
||||||
|
|
||||||
|
|
||||||
|
public HomeboxItemClient(RestClient.Builder restClienBuilder,
|
||||||
|
ConnectionRepository cRepository) {
|
||||||
|
this.restClientBuilder = restClienBuilder;
|
||||||
|
this.cRepository = cRepository;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fetches one page of items from Homebox for the given connection.
|
||||||
|
*
|
||||||
|
* @param connection identifies which stored connection (app URL, username) to query
|
||||||
|
* @param query optional keyword filter; {@code null} returns all items for the page
|
||||||
|
* @param pageable page number and size to request
|
||||||
|
* @return the mapped page of items along with the resolved connection ID
|
||||||
|
* @throws com.vaessl.app.exception.ConnectionNotFoundException if no matching connection is stored
|
||||||
|
* @throws com.vaessl.app.exception.RemoteApiException if Homebox returns an empty response body
|
||||||
|
*/
|
||||||
|
public HomeboxItemPage hbResponse(ConnectionIdentifiable connection, String query,
|
||||||
|
Pageable pageable) {
|
||||||
|
ConnectionEntity entity =
|
||||||
|
cRepository.findByAppUrlAndUsername(connection.appUrl(), connection.username());
|
||||||
|
|
||||||
|
if (!(entity instanceof HomeboxConnectionEntity hbEntity)) {
|
||||||
|
throw new ConnectionNotFoundException();
|
||||||
|
}
|
||||||
|
|
||||||
|
HomeboxItemsResponse response = restClientBuilder.baseUrl(connection.appUrl()).build().get()
|
||||||
|
.uri(u -> u.path(HOMEBOX_QUERY_ALL_ITEMS.getValue()).queryParam("q", query)
|
||||||
|
.queryParam("page", pageable.getPageNumber() + 1)
|
||||||
|
.queryParam("pageSize", pageable.getPageSize()).build())
|
||||||
|
.headers(h -> h.setBearerAuth(hbEntity.getToken())).retrieve()
|
||||||
|
.body(HomeboxItemsResponse.class);
|
||||||
|
|
||||||
|
if (response == null) {
|
||||||
|
throw new RemoteApiException(connection.appUrl(), HOMEBOX_QUERY_ALL_ITEMS.getValue());
|
||||||
|
}
|
||||||
|
|
||||||
|
List<ServiceItem> items = response.items().stream().map(i -> {
|
||||||
|
String id = i.id();
|
||||||
|
String title = i.name();
|
||||||
|
String description = i.description();
|
||||||
|
|
||||||
|
HomeboxParent parent = i.parent();
|
||||||
|
Map<String, Object> extraData = new HashMap<>();
|
||||||
|
if (parent.name() != null && !parent.name().isBlank()) {
|
||||||
|
extraData.put("locationName", parent.name());
|
||||||
|
}
|
||||||
|
if (parent.description() != null && !parent.description().isBlank()) {
|
||||||
|
extraData.put("locationDescription", parent.description());
|
||||||
|
}
|
||||||
|
|
||||||
|
return new ServiceItem(id, title, description, extraData);
|
||||||
|
}).toList();
|
||||||
|
|
||||||
|
return new HomeboxItemPage(new PageImpl<>(items, pageable, response.total()),
|
||||||
|
hbEntity.getId());
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
public record HomeboxItemPage(Page<ServiceItem> page, Long connectionId) {
|
||||||
|
}
|
||||||
|
|
||||||
|
private record HomeboxItemsResponse(int page, int pageSize, int total,
|
||||||
|
List<HomeboxItem> items) {
|
||||||
|
}
|
||||||
|
|
||||||
|
private record HomeboxItem(String id, String name, String description, HomeboxParent parent) {
|
||||||
|
}
|
||||||
|
|
||||||
|
private record HomeboxParent(String name, String description) {
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
}
|
||||||
@@ -1,36 +1,19 @@
|
|||||||
package com.vaessl.app.search;
|
package com.vaessl.app.search;
|
||||||
|
|
||||||
import java.util.List;
|
|
||||||
import java.util.Map;
|
|
||||||
|
|
||||||
import org.springframework.data.domain.Page;
|
import org.springframework.data.domain.Page;
|
||||||
import org.springframework.data.domain.PageImpl;
|
|
||||||
import org.springframework.data.domain.Pageable;
|
import org.springframework.data.domain.Pageable;
|
||||||
import org.springframework.stereotype.Component;
|
import org.springframework.stereotype.Component;
|
||||||
import org.springframework.web.client.RestClient;
|
|
||||||
|
|
||||||
import com.vaessl.app.connection.ConnectionEntity;
|
import com.vaessl.app.homebox.HomeboxItemClient;
|
||||||
import com.vaessl.app.connection.ConnectionRepository;
|
|
||||||
import com.vaessl.app.connection.HomeboxConnectionEntity;
|
|
||||||
import com.vaessl.app.exception.ConnectionNotFoundException;
|
|
||||||
import com.vaessl.app.exception.RemoteApiException;
|
|
||||||
import com.vaessl.app.shared.ServiceItem;
|
import com.vaessl.app.shared.ServiceItem;
|
||||||
import com.vaessl.app.shared.ServiceType;
|
import com.vaessl.app.shared.ServiceType;
|
||||||
|
import lombok.RequiredArgsConstructor;
|
||||||
import static com.vaessl.app.shared.Endpoint.*;
|
|
||||||
|
|
||||||
@Component
|
@Component
|
||||||
|
@RequiredArgsConstructor
|
||||||
public class HomeboxSearchProvider implements SearchProvider {
|
public class HomeboxSearchProvider implements SearchProvider {
|
||||||
|
|
||||||
private final RestClient.Builder restClientBuilder;
|
private final HomeboxItemClient client;
|
||||||
|
|
||||||
private final ConnectionRepository cRepository;
|
|
||||||
|
|
||||||
public HomeboxSearchProvider(RestClient.Builder restClientBuilder,
|
|
||||||
ConnectionRepository cRepository) {
|
|
||||||
this.restClientBuilder = restClientBuilder;
|
|
||||||
this.cRepository = cRepository;
|
|
||||||
}
|
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public ServiceType getServiceType() {
|
public ServiceType getServiceType() {
|
||||||
@@ -39,45 +22,6 @@ public class HomeboxSearchProvider implements SearchProvider {
|
|||||||
|
|
||||||
@Override
|
@Override
|
||||||
public Page<ServiceItem> getSearchResults(SearchRequest request, Pageable pageable) {
|
public Page<ServiceItem> getSearchResults(SearchRequest request, Pageable pageable) {
|
||||||
|
return client.hbResponse(request, request.query(), pageable).page();
|
||||||
ConnectionEntity entity =
|
|
||||||
cRepository.findByAppUrlAndUsername(request.appUrl(), request.username());
|
|
||||||
|
|
||||||
if (!(entity instanceof HomeboxConnectionEntity hbEntity)) {
|
|
||||||
throw new ConnectionNotFoundException();
|
|
||||||
}
|
|
||||||
|
|
||||||
HomeboxSearchResponse hbResponse = restClientBuilder.baseUrl(request.appUrl()).build().get()
|
|
||||||
.uri(u -> u.path(HOMEBOX_QUERY_ALL_ITEMS.getValue())
|
|
||||||
.queryParam("q", request.query())
|
|
||||||
.queryParam("page", pageable.getPageNumber() + 1)
|
|
||||||
.queryParam("pageSize", pageable.getPageSize()).build())
|
|
||||||
.headers(h -> h.setBearerAuth(hbEntity.getToken())).retrieve()
|
|
||||||
.body(HomeboxSearchResponse.class);
|
|
||||||
|
|
||||||
if (hbResponse == null) {
|
|
||||||
throw new RemoteApiException(request.appUrl(), HOMEBOX_QUERY_ALL_ITEMS.getValue());
|
|
||||||
}
|
|
||||||
|
|
||||||
List<ServiceItem> items = hbResponse.items().stream().map(i -> {
|
|
||||||
String id = i.id();
|
|
||||||
String title = i.name();
|
|
||||||
String description = i.description();
|
|
||||||
Map<String, Object> extraSearchResponseData = Map.of("parent", i.parent());
|
|
||||||
return new ServiceItem(id, title, description, extraSearchResponseData);
|
|
||||||
}).toList();
|
|
||||||
|
|
||||||
return new PageImpl<>(items, pageable, hbResponse.total());
|
|
||||||
}
|
|
||||||
|
|
||||||
private record HomeboxSearchResponse(int page, int pageSize, int total,
|
|
||||||
List<HomeboxItem> items) {
|
|
||||||
}
|
|
||||||
|
|
||||||
private record HomeboxItem(String id, String name, String description,
|
|
||||||
HomeboxExtraData parent) {
|
|
||||||
}
|
|
||||||
|
|
||||||
private record HomeboxExtraData(String name, String description) {
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,9 +1,10 @@
|
|||||||
package com.vaessl.app.search;
|
package com.vaessl.app.search;
|
||||||
|
|
||||||
|
import com.vaessl.app.connection.ConnectionIdentifiable;
|
||||||
import com.vaessl.app.shared.ServiceType;
|
import com.vaessl.app.shared.ServiceType;
|
||||||
import jakarta.validation.constraints.NotBlank;
|
import jakarta.validation.constraints.NotBlank;
|
||||||
import jakarta.validation.constraints.NotNull;
|
import jakarta.validation.constraints.NotNull;
|
||||||
|
|
||||||
public record SearchRequest(@NotBlank String appUrl, @NotBlank String username, String query,
|
public record SearchRequest(@NotBlank String appUrl, @NotBlank String username, String query,
|
||||||
@NotNull ServiceType serviceType, boolean aiSearch) {
|
@NotNull ServiceType serviceType, boolean aiSearch) implements ConnectionIdentifiable {
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,27 +1,57 @@
|
|||||||
package com.vaessl.app.sync;
|
package com.vaessl.app.sync;
|
||||||
|
|
||||||
import org.springframework.stereotype.Service;
|
import java.util.ArrayList;
|
||||||
|
import java.util.List;
|
||||||
|
import org.springframework.data.domain.Page;
|
||||||
|
import org.springframework.data.domain.PageRequest;
|
||||||
|
import org.springframework.data.domain.Pageable;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
import com.vaessl.app.homebox.HomeboxItemClient;
|
||||||
|
import com.vaessl.app.homebox.HomeboxItemClient.HomeboxItemPage;
|
||||||
|
import com.vaessl.app.shared.ServiceItem;
|
||||||
import com.vaessl.app.shared.ServiceType;
|
import com.vaessl.app.shared.ServiceType;
|
||||||
import com.vaessl.app.vector.EmbeddingService;
|
import com.vaessl.app.vector.EmbeddingService;
|
||||||
|
import lombok.RequiredArgsConstructor;
|
||||||
|
|
||||||
@Service
|
/**
|
||||||
|
* Pages through the full Homebox catalog and re-indexes it into the vector store.
|
||||||
|
* Item IDs are collected across all pages before {@link EmbeddingService#deleteStaleVectorEntries}
|
||||||
|
* is called once at the end, rather than after each page: deleting per page would treat every
|
||||||
|
* item outside the current page as stale, wiping out entries from pages already synced.
|
||||||
|
*/
|
||||||
|
@Component
|
||||||
|
@RequiredArgsConstructor
|
||||||
public class HomeboxSyncProvider implements SyncProvider {
|
public class HomeboxSyncProvider implements SyncProvider {
|
||||||
|
|
||||||
private final EmbeddingService embeddingService;
|
private final EmbeddingService embeddingService;
|
||||||
|
|
||||||
public HomeboxSyncProvider(EmbeddingService embeddingService) {
|
private final HomeboxItemClient client;
|
||||||
this.embeddingService = embeddingService;
|
|
||||||
}
|
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public ServiceType getServiceType() {
|
public ServiceType getServiceType() {
|
||||||
return ServiceType.HOMEBOX;
|
return ServiceType.HOMEBOX;
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public void syncVectorStore() {
|
public void syncVectorStore(SyncRequest request) {
|
||||||
// TODO Auto-generated method stub
|
|
||||||
throw new UnsupportedOperationException("Unimplemented method 'syncVectorStore'");
|
|
||||||
}
|
|
||||||
|
|
||||||
|
int batchSize = 100;
|
||||||
|
int page = 0;
|
||||||
|
Page<ServiceItem> current;
|
||||||
|
Long connectionId = null;
|
||||||
|
List<String> currentItemIds = new ArrayList<>();
|
||||||
|
do {
|
||||||
|
Pageable pageable = PageRequest.of(page, batchSize);
|
||||||
|
HomeboxItemPage result = client.hbResponse(request, null, pageable);
|
||||||
|
current = result.page();
|
||||||
|
if (connectionId == null) {
|
||||||
|
connectionId = result.connectionId();
|
||||||
|
}
|
||||||
|
embeddingService.vectorizeData(current.getContent(), request.serviceType(),
|
||||||
|
result.connectionId(), currentItemIds);
|
||||||
|
page++;
|
||||||
|
} while (page < current.getTotalPages());
|
||||||
|
|
||||||
|
embeddingService.deleteStaleVectorEntries(connectionId, currentItemIds);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,40 @@
|
|||||||
|
package com.vaessl.app.sync;
|
||||||
|
|
||||||
|
import org.springframework.http.HttpStatus;
|
||||||
|
import org.springframework.http.ResponseEntity;
|
||||||
|
import org.springframework.web.bind.annotation.PostMapping;
|
||||||
|
import org.springframework.web.bind.annotation.RequestBody;
|
||||||
|
import org.springframework.web.bind.annotation.RestController;
|
||||||
|
import com.vaessl.app.shared.SessionKeys;
|
||||||
|
import jakarta.servlet.http.HttpServletRequest;
|
||||||
|
import jakarta.servlet.http.HttpSession;
|
||||||
|
import jakarta.validation.Valid;
|
||||||
|
|
||||||
|
@RestController
|
||||||
|
public class SyncController {
|
||||||
|
|
||||||
|
private final SyncService syncService;
|
||||||
|
|
||||||
|
public SyncController(SyncService syncService) {
|
||||||
|
this.syncService = syncService;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Triggers a vector-store sync for the requested service. Returns {@code 401 Unauthorized}
|
||||||
|
* if there is no active session.
|
||||||
|
*/
|
||||||
|
@PostMapping("/sync")
|
||||||
|
public ResponseEntity<Object> syncVectorStore(@Valid @RequestBody SyncRequest request,
|
||||||
|
HttpServletRequest httpReq) {
|
||||||
|
|
||||||
|
HttpSession session = httpReq.getSession(false);
|
||||||
|
|
||||||
|
if (session == null
|
||||||
|
|| session.getAttribute(SessionKeys.connectionId(request.serviceType())) == null) {
|
||||||
|
return ResponseEntity.status(HttpStatus.UNAUTHORIZED).build();
|
||||||
|
}
|
||||||
|
syncService.syncServiceVectorStore(request);
|
||||||
|
|
||||||
|
return ResponseEntity.noContent().build();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -2,6 +2,16 @@ package com.vaessl.app.sync;
|
|||||||
|
|
||||||
import com.vaessl.app.shared.ServiceProvider;
|
import com.vaessl.app.shared.ServiceProvider;
|
||||||
|
|
||||||
public interface SyncProvider extends ServiceProvider{
|
/**
|
||||||
public void syncVectorStore();
|
* Implemented by any service that supports indexing its items into the vector store.
|
||||||
|
*/
|
||||||
|
public interface SyncProvider extends ServiceProvider {
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fetches all items from the remote service and indexes them into the vector store,
|
||||||
|
* removing any previously indexed items that no longer exist upstream.
|
||||||
|
*
|
||||||
|
* @param request the sync request containing the app URL and user credentials
|
||||||
|
*/
|
||||||
|
public void syncVectorStore(SyncRequest request);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,10 @@
|
|||||||
|
package com.vaessl.app.sync;
|
||||||
|
|
||||||
|
import com.vaessl.app.connection.ConnectionIdentifiable;
|
||||||
|
import com.vaessl.app.shared.ServiceType;
|
||||||
|
import jakarta.validation.constraints.NotBlank;
|
||||||
|
import jakarta.validation.constraints.NotNull;
|
||||||
|
|
||||||
|
public record SyncRequest(@NotBlank String appUrl, @NotBlank String username,
|
||||||
|
@NotNull ServiceType serviceType) implements ConnectionIdentifiable {
|
||||||
|
}
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
package com.vaessl.app.sync;
|
||||||
|
|
||||||
|
import java.util.EnumMap;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import org.springframework.stereotype.Service;
|
||||||
|
import com.vaessl.app.exception.WrongServiceTypeException;
|
||||||
|
import com.vaessl.app.shared.ServiceType;
|
||||||
|
|
||||||
|
@Service
|
||||||
|
public class SyncService {
|
||||||
|
|
||||||
|
private final Map<ServiceType, SyncProvider> providerRegistry;
|
||||||
|
|
||||||
|
public SyncService(List<SyncProvider> providers) {
|
||||||
|
Map<ServiceType, SyncProvider> registry = new EnumMap<>(ServiceType.class);
|
||||||
|
for (SyncProvider provider : providers) {
|
||||||
|
registry.put(provider.getServiceType(), provider);
|
||||||
|
}
|
||||||
|
this.providerRegistry = registry;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Dispatches the vector-store sync request to the provider registered for
|
||||||
|
* {@link SyncRequest#serviceType()}.
|
||||||
|
*
|
||||||
|
* @param request the sync request
|
||||||
|
* @throws WrongServiceTypeException if no provider is registered for the given service type
|
||||||
|
*/
|
||||||
|
public void syncServiceVectorStore(SyncRequest request) {
|
||||||
|
|
||||||
|
SyncProvider provider = providerRegistry.get(request.serviceType());
|
||||||
|
|
||||||
|
if (provider == null) {
|
||||||
|
throw new WrongServiceTypeException();
|
||||||
|
}
|
||||||
|
|
||||||
|
provider.syncVectorStore(request);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -6,10 +6,16 @@ import java.util.List;
|
|||||||
import java.util.Map;
|
import java.util.Map;
|
||||||
import org.springframework.ai.document.Document;
|
import org.springframework.ai.document.Document;
|
||||||
import org.springframework.ai.vectorstore.VectorStore;
|
import org.springframework.ai.vectorstore.VectorStore;
|
||||||
|
import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder;
|
||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
import com.vaessl.app.shared.ServiceItem;
|
import com.vaessl.app.shared.ServiceItem;
|
||||||
import com.vaessl.app.shared.ServiceType;
|
import com.vaessl.app.shared.ServiceType;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Embeds {@link ServiceItem}s into the shared vector store and prunes entries that no longer
|
||||||
|
* exist upstream. Kept stateless (no instance fields besides {@code vectorStore}) since this is
|
||||||
|
* a singleton bean shared across concurrent sync runs; callers own the per-sync ID accumulator.
|
||||||
|
*/
|
||||||
@Service
|
@Service
|
||||||
public class EmbeddingService {
|
public class EmbeddingService {
|
||||||
|
|
||||||
@@ -21,17 +27,31 @@ public class EmbeddingService {
|
|||||||
this.vectorStore = vectorStore;
|
this.vectorStore = vectorStore;
|
||||||
}
|
}
|
||||||
|
|
||||||
public void vectorizeData(List<ServiceItem> items, ServiceType serviceType, Long connectionId) {
|
/**
|
||||||
|
* Embeds and upserts {@code items} into the vector store, appending each item's ID to
|
||||||
|
* {@code currentItemIds} so the caller can later prune stale entries via
|
||||||
|
* {@link #deleteStaleVectorEntries}. Safe to call repeatedly (e.g. once per page) against the
|
||||||
|
* same accumulator across a single sync run.
|
||||||
|
*
|
||||||
|
* @param items the items to embed for this batch
|
||||||
|
* @param serviceType the originating service, stored in each document's metadata
|
||||||
|
* @param connectionId the connection these items belong to
|
||||||
|
* @param currentItemIds accumulator collecting every item ID seen so far in this sync run
|
||||||
|
*/
|
||||||
|
public void vectorizeData(List<ServiceItem> items, ServiceType serviceType, Long connectionId,
|
||||||
|
List<String> currentItemIds) {
|
||||||
List<Document> documents = new ArrayList<>();
|
List<Document> documents = new ArrayList<>();
|
||||||
|
|
||||||
for (ServiceItem item : items) {
|
for (ServiceItem item : items) {
|
||||||
documents.add(toDocument(item, serviceType, connectionId));
|
documents.add(toDocument(item, serviceType, connectionId));
|
||||||
|
currentItemIds.add(item.id());
|
||||||
}
|
}
|
||||||
vectorStore.add(documents);
|
vectorStore.add(documents);
|
||||||
}
|
}
|
||||||
|
|
||||||
private Document toDocument(ServiceItem item, ServiceType serviceType, Long connectionId) {
|
private Document toDocument(ServiceItem item, ServiceType serviceType, Long connectionId) {
|
||||||
String extraData = buildExtraData(item.extraData());
|
String extraData = buildExtraData(item.extraData());
|
||||||
Map<String, Object> metadata = buildMetadata(item, serviceType, connectionId, extraData);
|
Map<String, Object> metadata = buildMetadata(item, serviceType, connectionId);
|
||||||
String content = buildContent(item, extraData);
|
String content = buildContent(item, extraData);
|
||||||
return new Document(connectionId + ":" + item.id(), content, metadata);
|
return new Document(connectionId + ":" + item.id(), content, metadata);
|
||||||
}
|
}
|
||||||
@@ -54,30 +74,39 @@ public class EmbeddingService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private Map<String, Object> buildMetadata(ServiceItem item, ServiceType serviceType,
|
private Map<String, Object> buildMetadata(ServiceItem item, ServiceType serviceType,
|
||||||
Long connectionId, String extraData) {
|
Long connectionId) {
|
||||||
Map<String, Object> metadata = new HashMap<>();
|
Map<String, Object> metadata = new HashMap<>();
|
||||||
metadata.put("connectionId", connectionId);
|
metadata.put("connectionId", connectionId);
|
||||||
metadata.put("serviceType", serviceType.name());
|
metadata.put("serviceType", serviceType.name());
|
||||||
metadata.put("itemId", item.id());
|
metadata.put("itemId", item.id());
|
||||||
metadata.put("title", item.title());
|
|
||||||
if (item.description() != null && !item.description().isEmpty()) {
|
|
||||||
metadata.put("description", item.description());
|
|
||||||
}
|
|
||||||
if (!extraData.isEmpty()) {
|
|
||||||
metadata.put("extraData", extraData);
|
|
||||||
}
|
|
||||||
return metadata;
|
return metadata;
|
||||||
}
|
}
|
||||||
|
|
||||||
private String buildContent(ServiceItem item, String extraData) {
|
private String buildContent(ServiceItem item, String extraData) {
|
||||||
StringBuilder content = new StringBuilder("Title: ").append(item.title());
|
StringBuilder content = new StringBuilder("Title: ").append(item.title());
|
||||||
if (item.description() != null && !item.description().isEmpty()) {
|
if (item.description() != null && !item.description().isEmpty()) {
|
||||||
content.append("\n Description: ").append(item.description());
|
content.append("\nDescription: ").append(item.description());
|
||||||
}
|
}
|
||||||
if (!extraData.isEmpty()) {
|
if (!extraData.isEmpty()) {
|
||||||
content.append("\n Extradata: ").append(extraData);
|
content.append("\nExtradata: ").append(extraData);
|
||||||
}
|
}
|
||||||
return content.toString();
|
return content.toString();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Deletes every vector for {@code connectionId} whose item ID is not in
|
||||||
|
* {@code currentItemIds}, then clears the accumulator. Must be called once, after every page
|
||||||
|
* for this sync run has gone through {@link #vectorizeData}, not per page — otherwise items
|
||||||
|
* from pages other than the most recent one would look stale and get deleted too.
|
||||||
|
*
|
||||||
|
* @param connectionId the connection to prune stale vectors for
|
||||||
|
* @param currentItemIds every item ID seen across the full sync run; cleared after this call
|
||||||
|
*/
|
||||||
|
public void deleteStaleVectorEntries(Long connectionId, List<String> currentItemIds) {
|
||||||
|
FilterExpressionBuilder b = new FilterExpressionBuilder();
|
||||||
|
vectorStore.delete(b.and(b.eq("connectionId", connectionId),
|
||||||
|
b.nin("itemId", new ArrayList<>(currentItemIds))).build());
|
||||||
|
|
||||||
|
currentItemIds.clear();
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -10,8 +10,8 @@ import org.mockito.Mock;
|
|||||||
import org.mockito.junit.jupiter.MockitoExtension;
|
import org.mockito.junit.jupiter.MockitoExtension;
|
||||||
import org.springframework.data.domain.PageRequest;
|
import org.springframework.data.domain.PageRequest;
|
||||||
import org.springframework.data.domain.Pageable;
|
import org.springframework.data.domain.Pageable;
|
||||||
import com.vaessl.app.connection.ConnectionRepository;
|
|
||||||
import com.vaessl.app.exception.ConnectionNotFoundException;
|
import com.vaessl.app.exception.ConnectionNotFoundException;
|
||||||
|
import com.vaessl.app.homebox.HomeboxItemClient;
|
||||||
import static com.vaessl.app.shared.ServiceType.HOMEBOX;
|
import static com.vaessl.app.shared.ServiceType.HOMEBOX;
|
||||||
|
|
||||||
|
|
||||||
@@ -19,7 +19,7 @@ import static com.vaessl.app.shared.ServiceType.HOMEBOX;
|
|||||||
class HomeboxSearchProviderTest {
|
class HomeboxSearchProviderTest {
|
||||||
|
|
||||||
@Mock
|
@Mock
|
||||||
private ConnectionRepository mockRepo;
|
private HomeboxItemClient client;
|
||||||
|
|
||||||
@InjectMocks
|
@InjectMocks
|
||||||
private HomeboxSearchProvider provider;
|
private HomeboxSearchProvider provider;
|
||||||
@@ -27,11 +27,11 @@ class HomeboxSearchProviderTest {
|
|||||||
@Test
|
@Test
|
||||||
void shouldReturnConnectionNotFoundException() {
|
void shouldReturnConnectionNotFoundException() {
|
||||||
|
|
||||||
when(mockRepo.findByAppUrlAndUsername(MOCK_URL, MOCK_USER)).thenReturn(null);
|
|
||||||
|
|
||||||
SearchRequest request =
|
|
||||||
new SearchRequest(MOCK_URL, MOCK_USER, "test query", HOMEBOX, false);
|
|
||||||
Pageable pageable = PageRequest.of(0, 10);
|
Pageable pageable = PageRequest.of(0, 10);
|
||||||
|
SearchRequest request = new SearchRequest(MOCK_URL, MOCK_USER, "", HOMEBOX, false);
|
||||||
|
|
||||||
|
when(client.hbResponse(request, "", pageable)).thenThrow(new ConnectionNotFoundException());
|
||||||
|
|
||||||
assertThrows(ConnectionNotFoundException.class,
|
assertThrows(ConnectionNotFoundException.class,
|
||||||
() -> provider.getSearchResults(request, pageable));
|
() -> provider.getSearchResults(request, pageable));
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -93,7 +93,7 @@ class SearchControllerTest {
|
|||||||
.andExpect(status().isOk())
|
.andExpect(status().isOk())
|
||||||
.andExpect(jsonPath("$.content[0].title").value("MacBook Pro A1398"))
|
.andExpect(jsonPath("$.content[0].title").value("MacBook Pro A1398"))
|
||||||
.andExpect(jsonPath("$.totalElements").value(1))
|
.andExpect(jsonPath("$.totalElements").value(1))
|
||||||
.andExpect(jsonPath("$.content[0].extraData.parent.name").value("Server Schrank Ikea weiß"));
|
.andExpect(jsonPath("$.content[0].extraData.locationName").value("Server Schrank Ikea weiß"));
|
||||||
}
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
|
|||||||
Reference in New Issue
Block a user