Compare commits

..
Author SHA1 Message Date
kasun 8184db0c46 update CLAUDE.md to reflect the sync/vector package layout
Documents the homebox/, sync/, and vector/ packages as they actually
exist now (the doc still described a since-renamed ai/ package and a
VectorStoreConfig class that was never added - PgVectorStore is
auto-configured from application.yaml). Also flags what's still
pending rather than implying it's done: HomeboxAiSearchProvider is an
unimplemented stub, SearchService doesn't route on aiSearch yet, and
/sync isn't wired to login or the frontend.
2026-07-04 01:11:07 +02:00
kasun 65ff109800 add session-gated /sync endpoint and prune stale vectors after reindex
SyncController/SyncService/SyncProvider mirror the existing search
dispatch pattern: a session-gated controller routes a SyncRequest to
the provider registered for its ServiceType. HomeboxSyncProvider pages
through the full Homebox catalog, embedding each page via
EmbeddingService, and now collects every item ID it sees across all
pages before pruning anything no longer present upstream in one
deleteStaleVectorEntries call at the end - deleting per page would
treat items on other pages as stale and wipe them out.
2026-07-04 01:08:42 +02:00
kasun 48d120fbff extract HomeboxItemClient for reuse between search and sync
HomeboxSearchProvider used to own the RestClient call and item mapping
directly; pulling it into a shared HomeboxItemClient lets the upcoming
sync pipeline page through the same Homebox items without duplicating
the fetch/mapping logic. ConnectionIdentifiable lets both SearchRequest
and SyncRequest be resolved to a connection by the same client.
2026-07-04 01:08:26 +02:00
kasun 43b6d227db replaced constructor with RequiredArgsConstructor annotation 2026-07-01 19:47:00 +02:00
kasun fed981212a fixed failing searchControllerTests 2026-07-01 18:16:11 +02:00
kasun e310c1bbd8 renamed SyncProvider 2026-07-01 01:06:20 +02:00
kasun 9ae94c81e6 added vector and sync packages for pgvector operations, renamed classes and added sync providers. 2026-06-30 23:47:07 +02:00
kasun 8f24163bc0 refactored EmbeddingService 2026-06-30 23:26:33 +02:00
kasun 2ce90a6cac renamed Homebox record and extraData map key 2026-06-30 23:25:36 +02:00
kasun c363e16987 updated CLAUDE.md for better chat operations 2026-06-30 22:54:48 +02:00
kasun d5cf43ec08 renamed HomeboxEntity to HomeboxConnectionEntity 2026-06-30 22:31:34 +02:00
kasun 6908f7530a changed id type of vector store from uuid to text to make use of deterministic ids 2026-06-30 22:27:55 +02:00
kasun 573f0110c7 changed variable name 2026-06-30 22:25:41 +02:00
kasun 0508687cad added classes for vectorization, similarity search and summarization 2026-06-30 05:00:20 +02:00
kasun baa0582d50 renamed SearchResponse to ServiceItem in frontend 2026-06-30 04:55:08 +02:00
kasun 3086b7e2ca renamed SearchResponse to ServiceItem and moved to shared package 2026-06-30 04:52:45 +02:00
kasun 47466a6c61 added pgvector implementation and updated doc 2026-06-29 22:27:32 +02:00
kasun 28080c7954 dded postgresql dependency 2026-06-29 01:14:58 +02:00
kasun 23e4467e23 fixed format 2026-06-27 13:21:10 +02:00
29 changed files with 662 additions and 165 deletions
+39 -8
View File
@@ -49,13 +49,15 @@ Package-by-feature layout. Server context path is `/api`. Main endpoints:
- `GET /api/connections/status` — lists connected services for the current session - `GET /api/connections/status` — lists connected services for the current session
- `DELETE /api/connections/{serviceType}` — removes a service from the session; invalidates the session if no connections remain - `DELETE /api/connections/{serviceType}` — removes a service from the session; invalidates the session if no connections remain
- `POST /api/search` — paged search against the requested service; returns 401 if no active session - `POST /api/search` — paged search against the requested service; returns 401 if no active session
- `POST /api/sync` — triggers a vector-store sync for the requested service; returns 401 if no active session. Not yet triggered on login or called from the frontend — manual/internal trigger only for now.
Five packages: Eight packages:
**`shared/`** — cross-cutting types used by more than one feature package **`shared/`** — cross-cutting types used by more than one feature package
- `ServiceType` (enum): identifies each integrated app (e.g. `HOMEBOX`); used in both `connection/` and `search/` - `ServiceType` (enum): identifies each integrated app (e.g. `HOMEBOX`); used across `connection/`, `search/`, `sync/`
- `ServiceProvider` (interface): base for `ConnectionProvider` and `SearchProvider`; declares `getServiceType()` - `ServiceProvider` (interface): base for `ConnectionProvider`, `SearchProvider`, `SyncProvider`; declares `getServiceType()`
- `ServiceItem`: normalized item shape (`id`, `title`, `description`, `extraData`) returned by both search and sync fetches
- `Endpoint` (enum): API path constants for all external service calls - `Endpoint` (enum): API path constants for all external service calls
- `SessionKeys`: builds session attribute names of the form `{SERVICE_TYPE}_CONNECTION_ID` - `SessionKeys`: builds session attribute names of the form `{SERVICE_TYPE}_CONNECTION_ID`
@@ -67,17 +69,40 @@ Five packages:
**`connection/`** — connecting to and persisting service credentials **`connection/`** — connecting to and persisting service credentials
- `ConnectionProvider` interface: extends `ServiceProvider`; each integrated app implements `login()` and credential checking - `ConnectionProvider` interface: extends `ServiceProvider`; each integrated app implements `login()` and credential checking
- `ConnectionIdentifiable` interface: `appUrl()`/`username()`/`serviceType()`; implemented by `SearchRequest` and `SyncRequest` so `HomeboxItemClient` can resolve the underlying connection regardless of which feature is calling it
- `ConnectionService`: auto-discovers providers via Spring injection, dispatches login by `ServiceType` - `ConnectionService`: auto-discovers providers via Spring injection, dispatches login by `ServiceType`
- `ConnectionController`: stores `{serviceType}_CONNECTION_ID` in `HttpSession` after login; reads session attributes to build status responses - `ConnectionController`: stores `{serviceType}_CONNECTION_ID` in `HttpSession` after login; reads session attributes to build status responses
- Entity (`ConnectionEntity`) uses **Single Table Inheritance** — one `connections` table with app-specific nullable columns - Entity (`ConnectionEntity`) uses **Single Table Inheritance** — one `connections` table with app-specific nullable columns
- `HomeboxConnectionProvider` / `HomeboxEntity`: Homebox-specific implementation - `HomeboxConnectionProvider` / `HomeboxConnectionEntity`: Homebox-specific implementation
**`search/`** — querying connected services **`homebox/`** — shared Homebox API access
- `HomeboxItemClient`: fetches a page of items from the Homebox entities API and maps them to `ServiceItem`; used by both `HomeboxSearchProvider` (keyword search) and `HomeboxSyncProvider` (vector-store sync) so the fetch/mapping logic isn't duplicated
**`vector/`** — vectorization
- `EmbeddingService`: embeds `ServiceItem`s into pgvector's `VectorStore` and prunes entries that no longer exist upstream. Document ID is `{connectionId}:{itemId}`, so re-syncing upserts existing items instead of duplicating them. Deliberately stateless (only field is the injected `VectorStore`) since it's a singleton bean — the per-sync ID accumulator used for pruning is owned by the caller (`HomeboxSyncProvider`), not held as instance state
- `PgVectorStore` itself is auto-configured by `spring-ai-starter-vector-store-pgvector` from `application.yaml` (`spring.ai.vectorstore.pgvector.dimensions`) — no manual config class in this codebase
**`search/`** — querying connected services (keyword and AI)
- `SearchProvider` interface: extends `ServiceProvider`; each integrated app implements `getSearchResults()` - `SearchProvider` interface: extends `ServiceProvider`; each integrated app implements `getSearchResults()`
- `SearchService`: auto-discovers providers via Spring injection, dispatches by `ServiceType` - `SearchService`: dispatches by `ServiceType` via a single provider map — **does not yet route on `SearchRequest.aiSearch`**; `HomeboxSearchProvider` and `HomeboxAiSearchProvider` both currently register for `ServiceType.HOMEBOX`, so only one wins the registration (known gap, pending AI search work)
- `SearchController`: guards with session check before delegating to `SearchService` - `SearchController`: guards with session check before delegating to `SearchService`
- `HomeboxSearchProvider`: Homebox-specific search implementation using bearer token from session - `SearchRequest`: includes `aiSearch: boolean` (not yet consumed, see above) and implements `ConnectionIdentifiable`
- `PagedSearchResponse`: includes nullable `summary` field — populated only for AI search results; null for keyword search
- `HomeboxSearchProvider`: keyword search; delegates the remote fetch to `HomeboxItemClient`
- `HomeboxAiSearchProvider`: **stub only**`getSearchResults` throws `UnsupportedOperationException`; the actual similarity search + summary generation is pending
**`sync/`** — indexing connected services into the vector store
- `SyncProvider` interface: extends `ServiceProvider`; each integrated app implements `syncVectorStore()`
- `SyncService`: dispatches by `ServiceType` via a provider map, same pattern as `SearchService`
- `SyncController`: session-gated `POST /sync`; not currently called by the frontend or triggered on login
- `SyncRequest`: implements `ConnectionIdentifiable`
- `HomeboxSyncProvider`: pages through the full Homebox catalog via `HomeboxItemClient`, embedding each page through `EmbeddingService.vectorizeData` and collecting every item ID seen along the way. Once the full page loop finishes, calls `EmbeddingService.deleteStaleVectorEntries` **once** with the complete ID set, removing any previously indexed item no longer present upstream. Order matters — pruning per page instead of once at the end would treat items on other pages as stale and delete them too.
**Pending work (not yet implemented):** `HomeboxAiSearchProvider`'s actual similarity search + summary generation; `SearchService` routing two provider maps by `aiSearch`; triggering `/sync` on login or from the frontend; a frontend sync button and AI search UI.
**`exception/`** — `GlobalExceptionHandler` via `@ControllerAdvice` **`exception/`** — `GlobalExceptionHandler` via `@ControllerAdvice`
@@ -97,9 +122,15 @@ React 19 + TypeScript + SCSS, Vite 6 build. Package-by-feature under `components
### Data & AI ### Data & AI
- PostgreSQL + pgvector (semantic search via embeddings); also used as the Spring Session store (JDBC) - PostgreSQL + pgvector (semantic search via embeddings); also used as the Spring Session store (JDBC)
- LiteLLM as a unified AI proxy; Spring AI OpenAI starter wired to it - LiteLLM as a unified AI proxy; Spring AI OpenAI starter wired to it`OPENAI_BASE_URL` points to LiteLLM, not OpenAI directly, keeping the underlying model provider configurable
- `spring-ai-starter-vector-store-pgvector` provides `PgVectorStore`, auto-configured from `application.yaml` (no manual config class)
- Embedding dimensions must stay consistent with the configured LiteLLM embedding model — changing models requires re-syncing all indexed items
- Processing pipeline (Phase 2): stage in DB → LLM inference → refine via UI → export to target app - Processing pipeline (Phase 2): stage in DB → LLM inference → refine via UI → export to target app
### Testing Strategy ### Testing Strategy
Integration tests spin up a **mirrored PostgreSQL container** on port 5434 (same schema as production). WireMock mocks external HTTP APIs (Homebox, WikiJS). Do not mock the database in integration tests — the mirrored container strategy exists specifically to catch schema/migration divergence. Integration tests spin up a **mirrored PostgreSQL container** on port 5434 (same schema as production). WireMock mocks external HTTP APIs (Homebox, WikiJS). Do not mock the database in integration tests — the mirrored container strategy exists specifically to catch schema/migration divergence.
## Chat Operations
Don't make code suggestions and changes unless explicitly asked. Treat every prompt as a discussion of latest best practice coding approaches.
+5
View File
@@ -1,4 +1,5 @@
val wiremockVersion = "3.12.0" val wiremockVersion = "3.12.0"
val postgresqlVersion = "42.7.11"
plugins { plugins {
java java
@@ -46,6 +47,10 @@ dependencies {
implementation("org.springframework.boot:spring-boot-starter-validation") implementation("org.springframework.boot:spring-boot-starter-validation")
implementation("org.springframework.boot:spring-boot-starter-webmvc") implementation("org.springframework.boot:spring-boot-starter-webmvc")
implementation("org.springframework.ai:spring-ai-starter-model-openai") implementation("org.springframework.ai:spring-ai-starter-model-openai")
implementation("org.postgresql:postgresql:$postgresqlVersion")
implementation("org.springframework.ai:spring-ai-starter-vector-store-pgvector")
compileOnly("org.projectlombok:lombok") compileOnly("org.projectlombok:lombok")
@@ -0,0 +1,17 @@
package com.vaessl.app.connection;
import com.vaessl.app.shared.ServiceType;
/**
* Identifies the connection a request targets. Implemented by request records
* (e.g. {@code SearchRequest}, {@code SyncRequest}) so a single {@code HomeboxItemClient} can
* resolve the underlying connection regardless of which feature is calling it.
*/
public interface ConnectionIdentifiable {
String appUrl();
String username();
ServiceType serviceType();
}
@@ -11,15 +11,15 @@ import lombok.Setter;
@DiscriminatorValue("HOMEBOX") @DiscriminatorValue("HOMEBOX")
@Getter @Getter
@Setter @Setter
public class HomeboxEntity extends ConnectionEntity { public class HomeboxConnectionEntity extends ConnectionEntity {
private String token; private String token;
private String attachmentToken; private String attachmentToken;
private Instant expiresAt; private Instant expiresAt;
public static HomeboxEntity from(ConnectionRequest request, ConnectionResponse response) { public static HomeboxConnectionEntity from(ConnectionRequest request, ConnectionResponse response) {
HomeboxEntity he = new HomeboxEntity(); HomeboxConnectionEntity he = new HomeboxConnectionEntity();
he.setAppUrl(request.appUrl()); he.setAppUrl(request.appUrl());
he.setUsername(request.username()); he.setUsername(request.username());
@@ -12,22 +12,17 @@ import org.springframework.web.client.RestClient;
import com.vaessl.app.exception.EmptyCredentialsException; import com.vaessl.app.exception.EmptyCredentialsException;
import com.vaessl.app.exception.RemoteApiException; import com.vaessl.app.exception.RemoteApiException;
import com.vaessl.app.shared.ServiceType; import com.vaessl.app.shared.ServiceType;
import lombok.RequiredArgsConstructor;
import static com.vaessl.app.shared.Endpoint.*; import static com.vaessl.app.shared.Endpoint.*;
@Component @Component
@RequiredArgsConstructor
public class HomeboxConnectionProvider implements ConnectionProvider { public class HomeboxConnectionProvider implements ConnectionProvider {
private final RestClient.Builder restClientBuilder; private final RestClient.Builder restClientBuilder;
private final ConnectionRepository cRepository; private final ConnectionRepository cRepository;
public HomeboxConnectionProvider(RestClient.Builder restClientBuilder,
ConnectionRepository cRepository) {
this.restClientBuilder = restClientBuilder;
this.cRepository = cRepository;
}
@Override @Override
public void checkCredentials(ConnectionRequest request) { public void checkCredentials(ConnectionRequest request) {
if (request.username() == null || request.password() == null) { if (request.username() == null || request.password() == null) {
@@ -82,13 +77,13 @@ public class HomeboxConnectionProvider implements ConnectionProvider {
@Override @Override
public ConnectionEntity connectionToEntity(ConnectionRequest request, public ConnectionEntity connectionToEntity(ConnectionRequest request,
ConnectionResponse response) { ConnectionResponse response) {
return HomeboxEntity.from(request, response); return HomeboxConnectionEntity.from(request, response);
} }
@Override @Override
public void updateToRepository(ConnectionEntity existing, ConnectionResponse response) { public void updateToRepository(ConnectionEntity existing, ConnectionResponse response) {
if (existing instanceof HomeboxEntity hbE) { if (existing instanceof HomeboxConnectionEntity hbE) {
hbE.setToken(response.token()); hbE.setToken(response.token());
hbE.setExpiresAt(response.expiresAt()); hbE.setExpiresAt(response.expiresAt());
@@ -100,7 +95,7 @@ public class HomeboxConnectionProvider implements ConnectionProvider {
@Override @Override
public Instant getTokenExpiry(ConnectionEntity entity) { public Instant getTokenExpiry(ConnectionEntity entity) {
return (entity instanceof HomeboxEntity he) ? he.getExpiresAt() : null; return (entity instanceof HomeboxConnectionEntity he) ? he.getExpiresAt() : null;
} }
private record HomeboxLoginResponse(String token, String attachmentToken, Instant expiresAt) { private record HomeboxLoginResponse(String token, String attachmentToken, Instant expiresAt) {
@@ -0,0 +1,105 @@
package com.vaessl.app.homebox;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageImpl;
import org.springframework.data.domain.Pageable;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;
import com.vaessl.app.connection.ConnectionEntity;
import com.vaessl.app.connection.ConnectionIdentifiable;
import com.vaessl.app.connection.ConnectionRepository;
import com.vaessl.app.connection.HomeboxConnectionEntity;
import com.vaessl.app.exception.ConnectionNotFoundException;
import com.vaessl.app.exception.RemoteApiException;
import com.vaessl.app.shared.ServiceItem;
import static com.vaessl.app.shared.Endpoint.HOMEBOX_QUERY_ALL_ITEMS;
/**
* Fetches items from the Homebox entities API. Shared by {@code HomeboxSearchProvider} and
* {@code HomeboxSyncProvider} so both keyword search and vector-store sync page through the same
* remote call and item mapping.
*/
@Component
public class HomeboxItemClient {
private final RestClient.Builder restClientBuilder;
private final ConnectionRepository cRepository;
public HomeboxItemClient(RestClient.Builder restClienBuilder,
ConnectionRepository cRepository) {
this.restClientBuilder = restClienBuilder;
this.cRepository = cRepository;
}
/**
* Fetches one page of items from Homebox for the given connection.
*
* @param connection identifies which stored connection (app URL, username) to query
* @param query optional keyword filter; {@code null} returns all items for the page
* @param pageable page number and size to request
* @return the mapped page of items along with the resolved connection ID
* @throws com.vaessl.app.exception.ConnectionNotFoundException if no matching connection is stored
* @throws com.vaessl.app.exception.RemoteApiException if Homebox returns an empty response body
*/
public HomeboxItemPage hbResponse(ConnectionIdentifiable connection, String query,
Pageable pageable) {
ConnectionEntity entity =
cRepository.findByAppUrlAndUsername(connection.appUrl(), connection.username());
if (!(entity instanceof HomeboxConnectionEntity hbEntity)) {
throw new ConnectionNotFoundException();
}
HomeboxItemsResponse response = restClientBuilder.baseUrl(connection.appUrl()).build().get()
.uri(u -> u.path(HOMEBOX_QUERY_ALL_ITEMS.getValue()).queryParam("q", query)
.queryParam("page", pageable.getPageNumber() + 1)
.queryParam("pageSize", pageable.getPageSize()).build())
.headers(h -> h.setBearerAuth(hbEntity.getToken())).retrieve()
.body(HomeboxItemsResponse.class);
if (response == null) {
throw new RemoteApiException(connection.appUrl(), HOMEBOX_QUERY_ALL_ITEMS.getValue());
}
List<ServiceItem> items = response.items().stream().map(i -> {
String id = i.id();
String title = i.name();
String description = i.description();
HomeboxParent parent = i.parent();
Map<String, Object> extraData = new HashMap<>();
if (parent.name() != null && !parent.name().isBlank()) {
extraData.put("locationName", parent.name());
}
if (parent.description() != null && !parent.description().isBlank()) {
extraData.put("locationDescription", parent.description());
}
return new ServiceItem(id, title, description, extraData);
}).toList();
return new HomeboxItemPage(new PageImpl<>(items, pageable, response.total()),
hbEntity.getId());
}
public record HomeboxItemPage(Page<ServiceItem> page, Long connectionId) {
}
private record HomeboxItemsResponse(int page, int pageSize, int total,
List<HomeboxItem> items) {
}
private record HomeboxItem(String id, String name, String description, HomeboxParent parent) {
}
private record HomeboxParent(String name, String description) {
}
}
@@ -0,0 +1,22 @@
package com.vaessl.app.search;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable;
import org.springframework.stereotype.Component;
import com.vaessl.app.shared.ServiceItem;
import com.vaessl.app.shared.ServiceType;
@Component
public class HomeboxAiSearchProvider implements SearchProvider {
@Override
public ServiceType getServiceType() {
return ServiceType.HOMEBOX;
}
@Override
public Page<ServiceItem> getSearchResults(SearchRequest request, Pageable pageable) {
// TODO Auto-generated method stub
throw new UnsupportedOperationException("Unimplemented method 'getSearchResults'");
}
}
@@ -1,35 +1,19 @@
package com.vaessl.app.search; package com.vaessl.app.search;
import java.util.List;
import java.util.Map;
import org.springframework.data.domain.Page; import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageImpl;
import org.springframework.data.domain.Pageable; import org.springframework.data.domain.Pageable;
import org.springframework.stereotype.Component; import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;
import com.vaessl.app.connection.ConnectionEntity; import com.vaessl.app.homebox.HomeboxItemClient;
import com.vaessl.app.connection.ConnectionRepository; import com.vaessl.app.shared.ServiceItem;
import com.vaessl.app.connection.HomeboxEntity;
import com.vaessl.app.exception.ConnectionNotFoundException;
import com.vaessl.app.exception.RemoteApiException;
import com.vaessl.app.shared.ServiceType; import com.vaessl.app.shared.ServiceType;
import lombok.RequiredArgsConstructor;
import static com.vaessl.app.shared.Endpoint.*;
@Component @Component
@RequiredArgsConstructor
public class HomeboxSearchProvider implements SearchProvider { public class HomeboxSearchProvider implements SearchProvider {
private final RestClient.Builder restClientBuilder; private final HomeboxItemClient client;
private final ConnectionRepository cRepository;
public HomeboxSearchProvider(RestClient.Builder restClientBuilder,
ConnectionRepository cRepository) {
this.restClientBuilder = restClientBuilder;
this.cRepository = cRepository;
}
@Override @Override
public ServiceType getServiceType() { public ServiceType getServiceType() {
@@ -37,45 +21,7 @@ public class HomeboxSearchProvider implements SearchProvider {
} }
@Override @Override
public Page<SearchResponse> getSearchResults(SearchRequest request, Pageable pageable) { public Page<ServiceItem> getSearchResults(SearchRequest request, Pageable pageable) {
return client.hbResponse(request, request.query(), pageable).page();
ConnectionEntity entity =
cRepository.findByAppUrlAndUsername(request.appUrl(), request.username());
if (!(entity instanceof HomeboxEntity hbEntity)) {
throw new ConnectionNotFoundException();
}
HomeboxSearchResponse hbResponse = restClientBuilder.baseUrl(request.appUrl()).build().get()
.uri(u -> u.path(HOMEBOX_QUERY_ALL_ITEMS.getValue())
.queryParam("q", request.query())
.queryParam("page", pageable.getPageNumber() + 1)
.queryParam("pageSize", pageable.getPageSize()).build())
.headers(h -> h.setBearerAuth(hbEntity.getToken())).retrieve()
.body(HomeboxSearchResponse.class);
if (hbResponse == null) {
throw new RemoteApiException(request.appUrl(), HOMEBOX_QUERY_ALL_ITEMS.getValue());
}
List<SearchResponse> items = hbResponse.items().stream().map(i -> {
String id = i.id();
String title = i.name();
String description = i.description();
Map<String, Object> extraSearchResponseData = Map.of("location", i.parent());
return new SearchResponse(id, title, description, extraSearchResponseData);
}).toList();
return new PageImpl<>(items, pageable, hbResponse.total());
}
private record HomeboxSearchResponse(int page, int pageSize, int total,
List<HomeboxItem> items) {
}
private record HomeboxItem(String id, String name, String description, HomeboxLocation parent) {
}
private record HomeboxLocation(String name, String description) {
} }
} }
@@ -4,11 +4,11 @@ import java.util.List;
import org.springframework.data.domain.Page; import org.springframework.data.domain.Page;
public record PagedSearchResponse<T>(List<T> content, int page, int pageSize, long totalElements, public record PagedSearchResponse<T>(List<T> content, int page, int pageSize, long totalElements,
boolean first, boolean last, String sort) { boolean first, boolean last, String sort, String summary) {
public static <T> PagedSearchResponse<T> from(Page<T> pageResult) { public static <T> PagedSearchResponse<T> from(Page<T> pageResult) {
return new PagedSearchResponse<>(pageResult.getContent(), pageResult.getNumber(), return new PagedSearchResponse<>(pageResult.getContent(), pageResult.getNumber(),
pageResult.getSize(), pageResult.getTotalElements(), pageResult.isFirst(), pageResult.getSize(), pageResult.getTotalElements(), pageResult.isFirst(),
pageResult.isLast(), pageResult.getSort().toString()); pageResult.isLast(), pageResult.getSort().toString(), null);
} }
} }
@@ -8,7 +8,7 @@ import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody; import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController; import org.springframework.web.bind.annotation.RestController;
import com.vaessl.app.shared.ServiceItem;
import com.vaessl.app.shared.SessionKeys; import com.vaessl.app.shared.SessionKeys;
import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpSession; import jakarta.servlet.http.HttpSession;
@@ -26,7 +26,7 @@ public class SearchController {
* there is no active session. * there is no active session.
*/ */
@PostMapping("/search") @PostMapping("/search")
public ResponseEntity<PagedSearchResponse<SearchResponse>> search( public ResponseEntity<PagedSearchResponse<ServiceItem>> search(
@Valid @RequestBody SearchRequest request, @Valid @RequestBody SearchRequest request,
@PageableDefault(size = 20) Pageable pageable, HttpServletRequest httpReq) { @PageableDefault(size = 20) Pageable pageable, HttpServletRequest httpReq) {
HttpSession session = httpReq.getSession(false); HttpSession session = httpReq.getSession(false);
@@ -35,7 +35,7 @@ public class SearchController {
return ResponseEntity.status(HttpStatus.UNAUTHORIZED).build(); return ResponseEntity.status(HttpStatus.UNAUTHORIZED).build();
} }
Page<SearchResponse> result = searchService.search(request, pageable); Page<ServiceItem> result = searchService.search(request, pageable);
return ResponseEntity.ok(PagedSearchResponse.from(result)); return ResponseEntity.ok(PagedSearchResponse.from(result));
} }
@@ -2,7 +2,7 @@ package com.vaessl.app.search;
import org.springframework.data.domain.Page; import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable; import org.springframework.data.domain.Pageable;
import com.vaessl.app.shared.ServiceItem;
import com.vaessl.app.shared.ServiceProvider; import com.vaessl.app.shared.ServiceProvider;
/** /**
@@ -17,5 +17,5 @@ public interface SearchProvider extends ServiceProvider {
* @param pageable the Pageable interface * @param pageable the Pageable interface
* @return a list of Page<SearchResponse> items matching the query * @return a list of Page<SearchResponse> items matching the query
*/ */
Page<SearchResponse> getSearchResults(SearchRequest request, Pageable pageable); Page<ServiceItem> getSearchResults(SearchRequest request, Pageable pageable);
} }
@@ -1,9 +1,10 @@
package com.vaessl.app.search; package com.vaessl.app.search;
import com.vaessl.app.connection.ConnectionIdentifiable;
import com.vaessl.app.shared.ServiceType; import com.vaessl.app.shared.ServiceType;
import jakarta.validation.constraints.NotBlank; import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotNull; import jakarta.validation.constraints.NotNull;
public record SearchRequest(@NotBlank String appUrl, @NotBlank String username, String query, public record SearchRequest(@NotBlank String appUrl, @NotBlank String username, String query,
@NotNull ServiceType serviceType) { @NotNull ServiceType serviceType, boolean aiSearch) implements ConnectionIdentifiable {
} }
@@ -7,6 +7,7 @@ import java.util.Map;
import org.springframework.data.domain.Page; import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable; import org.springframework.data.domain.Pageable;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import com.vaessl.app.shared.ServiceItem;
import com.vaessl.app.shared.ServiceType; import com.vaessl.app.shared.ServiceType;
import com.vaessl.app.exception.WrongServiceTypeException; import com.vaessl.app.exception.WrongServiceTypeException;
@@ -21,7 +22,6 @@ public class SearchService {
registry.put(provider.getServiceType(), provider); registry.put(provider.getServiceType(), provider);
} }
this.providerRegistry = registry; this.providerRegistry = registry;
} }
/** /**
@@ -33,7 +33,7 @@ public class SearchService {
* @return results returned by the matching provider * @return results returned by the matching provider
* @throws WrongServiceTypeException if no provider is registered for the given service type * @throws WrongServiceTypeException if no provider is registered for the given service type
*/ */
public Page<SearchResponse> search(SearchRequest request, Pageable pageable) { public Page<ServiceItem> search(SearchRequest request, Pageable pageable) {
SearchProvider provider = providerRegistry.get(request.serviceType()); SearchProvider provider = providerRegistry.get(request.serviceType());
@@ -1,8 +1,9 @@
package com.vaessl.app.search; package com.vaessl.app.shared;
import java.util.Map; import java.util.Map;
import jakarta.validation.constraints.NotNull;
public record SearchResponse(String id, String title, String description, public record ServiceItem(String id, @NotNull String title, String description,
Map<String, Object> extraData) { Map<String, Object> extraData) {
public String getExtra(String key) { public String getExtra(String key) {
@@ -0,0 +1,57 @@
package com.vaessl.app.sync;
import java.util.ArrayList;
import java.util.List;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.domain.Pageable;
import org.springframework.stereotype.Component;
import com.vaessl.app.homebox.HomeboxItemClient;
import com.vaessl.app.homebox.HomeboxItemClient.HomeboxItemPage;
import com.vaessl.app.shared.ServiceItem;
import com.vaessl.app.shared.ServiceType;
import com.vaessl.app.vector.EmbeddingService;
import lombok.RequiredArgsConstructor;
/**
* Pages through the full Homebox catalog and re-indexes it into the vector store.
* Item IDs are collected across all pages before {@link EmbeddingService#deleteStaleVectorEntries}
* is called once at the end, rather than after each page: deleting per page would treat every
* item outside the current page as stale, wiping out entries from pages already synced.
*/
@Component
@RequiredArgsConstructor
public class HomeboxSyncProvider implements SyncProvider {
private final EmbeddingService embeddingService;
private final HomeboxItemClient client;
@Override
public ServiceType getServiceType() {
return ServiceType.HOMEBOX;
}
@Override
public void syncVectorStore(SyncRequest request) {
int batchSize = 100;
int page = 0;
Page<ServiceItem> current;
Long connectionId = null;
List<String> currentItemIds = new ArrayList<>();
do {
Pageable pageable = PageRequest.of(page, batchSize);
HomeboxItemPage result = client.hbResponse(request, null, pageable);
current = result.page();
if (connectionId == null) {
connectionId = result.connectionId();
}
embeddingService.vectorizeData(current.getContent(), request.serviceType(),
result.connectionId(), currentItemIds);
page++;
} while (page < current.getTotalPages());
embeddingService.deleteStaleVectorEntries(connectionId, currentItemIds);
}
}
@@ -0,0 +1,40 @@
package com.vaessl.app.sync;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;
import com.vaessl.app.shared.SessionKeys;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpSession;
import jakarta.validation.Valid;
@RestController
public class SyncController {
private final SyncService syncService;
public SyncController(SyncService syncService) {
this.syncService = syncService;
}
/**
* Triggers a vector-store sync for the requested service. Returns {@code 401 Unauthorized}
* if there is no active session.
*/
@PostMapping("/sync")
public ResponseEntity<Object> syncVectorStore(@Valid @RequestBody SyncRequest request,
HttpServletRequest httpReq) {
HttpSession session = httpReq.getSession(false);
if (session == null
|| session.getAttribute(SessionKeys.connectionId(request.serviceType())) == null) {
return ResponseEntity.status(HttpStatus.UNAUTHORIZED).build();
}
syncService.syncServiceVectorStore(request);
return ResponseEntity.noContent().build();
}
}
@@ -0,0 +1,17 @@
package com.vaessl.app.sync;
import com.vaessl.app.shared.ServiceProvider;
/**
* Implemented by any service that supports indexing its items into the vector store.
*/
public interface SyncProvider extends ServiceProvider {
/**
* Fetches all items from the remote service and indexes them into the vector store,
* removing any previously indexed items that no longer exist upstream.
*
* @param request the sync request containing the app URL and user credentials
*/
public void syncVectorStore(SyncRequest request);
}
@@ -0,0 +1,10 @@
package com.vaessl.app.sync;
import com.vaessl.app.connection.ConnectionIdentifiable;
import com.vaessl.app.shared.ServiceType;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotNull;
public record SyncRequest(@NotBlank String appUrl, @NotBlank String username,
@NotNull ServiceType serviceType) implements ConnectionIdentifiable {
}
@@ -0,0 +1,40 @@
package com.vaessl.app.sync;
import java.util.EnumMap;
import java.util.List;
import java.util.Map;
import org.springframework.stereotype.Service;
import com.vaessl.app.exception.WrongServiceTypeException;
import com.vaessl.app.shared.ServiceType;
@Service
public class SyncService {
private final Map<ServiceType, SyncProvider> providerRegistry;
public SyncService(List<SyncProvider> providers) {
Map<ServiceType, SyncProvider> registry = new EnumMap<>(ServiceType.class);
for (SyncProvider provider : providers) {
registry.put(provider.getServiceType(), provider);
}
this.providerRegistry = registry;
}
/**
* Dispatches the vector-store sync request to the provider registered for
* {@link SyncRequest#serviceType()}.
*
* @param request the sync request
* @throws WrongServiceTypeException if no provider is registered for the given service type
*/
public void syncServiceVectorStore(SyncRequest request) {
SyncProvider provider = providerRegistry.get(request.serviceType());
if (provider == null) {
throw new WrongServiceTypeException();
}
provider.syncVectorStore(request);
}
}
@@ -0,0 +1,112 @@
package com.vaessl.app.vector;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.springframework.ai.document.Document;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder;
import org.springframework.stereotype.Service;
import com.vaessl.app.shared.ServiceItem;
import com.vaessl.app.shared.ServiceType;
/**
* Embeds {@link ServiceItem}s into the shared vector store and prunes entries that no longer
* exist upstream. Kept stateless (no instance fields besides {@code vectorStore}) since this is
* a singleton bean shared across concurrent sync runs; callers own the per-sync ID accumulator.
*/
@Service
public class EmbeddingService {
private static final float THRESHOLD = 0.7f;
private static final int LIMIT = 2;
private final VectorStore vectorStore;
public EmbeddingService(VectorStore vectorStore) {
this.vectorStore = vectorStore;
}
/**
* Embeds and upserts {@code items} into the vector store, appending each item's ID to
* {@code currentItemIds} so the caller can later prune stale entries via
* {@link #deleteStaleVectorEntries}. Safe to call repeatedly (e.g. once per page) against the
* same accumulator across a single sync run.
*
* @param items the items to embed for this batch
* @param serviceType the originating service, stored in each document's metadata
* @param connectionId the connection these items belong to
* @param currentItemIds accumulator collecting every item ID seen so far in this sync run
*/
public void vectorizeData(List<ServiceItem> items, ServiceType serviceType, Long connectionId,
List<String> currentItemIds) {
List<Document> documents = new ArrayList<>();
for (ServiceItem item : items) {
documents.add(toDocument(item, serviceType, connectionId));
currentItemIds.add(item.id());
}
vectorStore.add(documents);
}
private Document toDocument(ServiceItem item, ServiceType serviceType, Long connectionId) {
String extraData = buildExtraData(item.extraData());
Map<String, Object> metadata = buildMetadata(item, serviceType, connectionId);
String content = buildContent(item, extraData);
return new Document(connectionId + ":" + item.id(), content, metadata);
}
private String buildExtraData(Map<String, Object> extraData) {
if (extraData == null) {
return "";
}
StringBuilder data = new StringBuilder();
for (Map.Entry<String, Object> entry : extraData.entrySet()) {
if (entry.getValue() == null || entry.getValue().toString().isEmpty()) {
continue;
}
if (!data.isEmpty()) {
data.append(" \n");
}
data.append(entry.getKey()).append(": ").append(entry.getValue());
}
return data.toString();
}
private Map<String, Object> buildMetadata(ServiceItem item, ServiceType serviceType,
Long connectionId) {
Map<String, Object> metadata = new HashMap<>();
metadata.put("connectionId", connectionId);
metadata.put("serviceType", serviceType.name());
metadata.put("itemId", item.id());
return metadata;
}
private String buildContent(ServiceItem item, String extraData) {
StringBuilder content = new StringBuilder("Title: ").append(item.title());
if (item.description() != null && !item.description().isEmpty()) {
content.append("\nDescription: ").append(item.description());
}
if (!extraData.isEmpty()) {
content.append("\nExtradata: ").append(extraData);
}
return content.toString();
}
/**
* Deletes every vector for {@code connectionId} whose item ID is not in
* {@code currentItemIds}, then clears the accumulator. Must be called once, after every page
* for this sync run has gone through {@link #vectorizeData}, not per page — otherwise items
* from pages other than the most recent one would look stale and get deleted too.
*
* @param connectionId the connection to prune stale vectors for
* @param currentItemIds every item ID seen across the full sync run; cleared after this call
*/
public void deleteStaleVectorEntries(Long connectionId, List<String> currentItemIds) {
FilterExpressionBuilder b = new FilterExpressionBuilder();
vectorStore.delete(b.and(b.eq("connectionId", connectionId),
b.nin("itemId", new ArrayList<>(currentItemIds))).build());
currentItemIds.clear();
}
}
@@ -21,6 +21,13 @@ spring:
api-key: ${OPENAI_KEY} api-key: ${OPENAI_KEY}
chat: chat:
model: gpt-4o-mini model: gpt-4o-mini
vectorstore:
pgvector:
id-type: text
dimensions: 1536
distance-type: COSINE_DISTANCE
index-type: HNSW
initialize-schema: true
session: session:
store-type: jdbc store-type: jdbc
jdbc: jdbc:
@@ -180,7 +180,7 @@ class HomeboxIntegrationTest {
assertThat(dbEntry.getAppUrl()).isEqualTo(request.appUrl()); assertThat(dbEntry.getAppUrl()).isEqualTo(request.appUrl());
assertThat(dbEntry.getUsername()).isEqualTo(request.username()); assertThat(dbEntry.getUsername()).isEqualTo(request.username());
if (dbEntry instanceof HomeboxEntity hbE) { if (dbEntry instanceof HomeboxConnectionEntity hbE) {
assertThat(hbE.getToken()).isEqualTo("fake-jwt-token"); assertThat(hbE.getToken()).isEqualTo("fake-jwt-token");
assertThat(hbE.getAttachmentToken()).isEqualTo("fake-attach"); assertThat(hbE.getAttachmentToken()).isEqualTo("fake-attach");
assertThat(hbE.getExpiresAt().toString()) assertThat(hbE.getExpiresAt().toString())
@@ -10,8 +10,8 @@ import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension; import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.data.domain.PageRequest; import org.springframework.data.domain.PageRequest;
import org.springframework.data.domain.Pageable; import org.springframework.data.domain.Pageable;
import com.vaessl.app.connection.ConnectionRepository;
import com.vaessl.app.exception.ConnectionNotFoundException; import com.vaessl.app.exception.ConnectionNotFoundException;
import com.vaessl.app.homebox.HomeboxItemClient;
import static com.vaessl.app.shared.ServiceType.HOMEBOX; import static com.vaessl.app.shared.ServiceType.HOMEBOX;
@@ -19,7 +19,7 @@ import static com.vaessl.app.shared.ServiceType.HOMEBOX;
class HomeboxSearchProviderTest { class HomeboxSearchProviderTest {
@Mock @Mock
private ConnectionRepository mockRepo; private HomeboxItemClient client;
@InjectMocks @InjectMocks
private HomeboxSearchProvider provider; private HomeboxSearchProvider provider;
@@ -27,10 +27,11 @@ class HomeboxSearchProviderTest {
@Test @Test
void shouldReturnConnectionNotFoundException() { void shouldReturnConnectionNotFoundException() {
when(mockRepo.findByAppUrlAndUsername(MOCK_URL, MOCK_USER)).thenReturn(null);
SearchRequest request = new SearchRequest(MOCK_URL, MOCK_USER, "test query", HOMEBOX);
Pageable pageable = PageRequest.of(0, 10); Pageable pageable = PageRequest.of(0, 10);
SearchRequest request = new SearchRequest(MOCK_URL, MOCK_USER, "", HOMEBOX, false);
when(client.hbResponse(request, "", pageable)).thenThrow(new ConnectionNotFoundException());
assertThrows(ConnectionNotFoundException.class, assertThrows(ConnectionNotFoundException.class,
() -> provider.getSearchResults(request, pageable)); () -> provider.getSearchResults(request, pageable));
} }
@@ -93,8 +93,7 @@ class SearchControllerTest {
.andExpect(status().isOk()) .andExpect(status().isOk())
.andExpect(jsonPath("$.content[0].title").value("MacBook Pro A1398")) .andExpect(jsonPath("$.content[0].title").value("MacBook Pro A1398"))
.andExpect(jsonPath("$.totalElements").value(1)) .andExpect(jsonPath("$.totalElements").value(1))
.andExpect(jsonPath("$.content[0].extraData.location.name") .andExpect(jsonPath("$.content[0].extraData.locationName").value("Server Schrank Ikea weiß"));
.value("Server Schrank Ikea weiß"));
} }
@Test @Test
@@ -104,7 +103,8 @@ class SearchControllerTest {
"appUrl": "http://irrelevant", "appUrl": "http://irrelevant",
"query": "Item", "query": "Item",
"serviceType": "HOMEBOX", "serviceType": "HOMEBOX",
"username": "irrelevant" "username": "irrelevant",
"aiSearch": false
} }
""")).andExpect(status().isUnauthorized()); """)).andExpect(status().isUnauthorized());
} }
@@ -117,22 +117,23 @@ class SearchControllerTest {
} }
private String searchRequestBody(WireMockRuntimeInfo wm, String serviceType) { private String searchRequestBody(WireMockRuntimeInfo wm, String serviceType) {
return searchRequestBody(wm.getHttpBaseUrl(), serviceType); return searchRequestBody(wm.getHttpBaseUrl(), serviceType, false);
} }
private String searchRequestBody(String serviceType) { private String searchRequestBody(String serviceType) {
return searchRequestBody("http://irrelevant", serviceType); return searchRequestBody("http://irrelevant", serviceType, false);
} }
private String searchRequestBody(String appUrl, String serviceType) { private String searchRequestBody(String appUrl, String serviceType, boolean aiSearch) {
return """ return """
{ {
"appUrl": "%s", "appUrl": "%s",
"query": "Item", "query": "Item",
"serviceType": "%s", "serviceType": "%s",
"username": "%s" "username": "%s",
"aiSearch": "%b"
} }
""".formatted(appUrl, serviceType, MOCK_USER); """.formatted(appUrl, serviceType, MOCK_USER, aiSearch);
} }
private String connectionRequestBody(WireMockRuntimeInfo wm) { private String connectionRequestBody(WireMockRuntimeInfo wm) {
@@ -4,12 +4,13 @@ import static com.vaessl.app.Mockdata.*;
import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThat;
import java.util.Map; import java.util.Map;
import org.junit.jupiter.api.Test; import org.junit.jupiter.api.Test;
import com.vaessl.app.shared.ServiceItem;
class SearchResponseTest { class SearchResponseTest {
@Test @Test
void shouldReturnNullWhenExtraDataIsNull() { void shouldReturnNullWhenExtraDataIsNull() {
SearchResponse response = new SearchResponse(MOCK_ID, MOCK_TITLE, MOCK_DESCRIPTION, null); ServiceItem response = new ServiceItem(MOCK_ID, MOCK_TITLE, MOCK_DESCRIPTION, null);
assertThat(response.getExtra(null)).isNull(); assertThat(response.getExtra(null)).isNull();
} }
@@ -17,7 +18,7 @@ class SearchResponseTest {
@Test @Test
void shouldReturnNullWhenExtraDataKeyIsMissing() { void shouldReturnNullWhenExtraDataKeyIsMissing() {
SearchResponse response = new SearchResponse(MOCK_ID, MOCK_TITLE, MOCK_DESCRIPTION, Map.of("key", "value")); ServiceItem response = new ServiceItem(MOCK_ID, MOCK_TITLE, MOCK_DESCRIPTION, Map.of("key", "value"));
assertThat(response.getExtra("missing")).isNull(); assertThat(response.getExtra("missing")).isNull();
} }
@@ -25,7 +26,7 @@ class SearchResponseTest {
@Test @Test
void shouldReturnExtraDataValue() { void shouldReturnExtraDataValue() {
SearchResponse response = new SearchResponse(MOCK_ID, MOCK_TITLE, MOCK_DESCRIPTION, Map.of("key", "value")); ServiceItem response = new ServiceItem(MOCK_ID, MOCK_TITLE, MOCK_DESCRIPTION, Map.of("key", "value"));
assertThat(response.id()).isEqualTo(MOCK_ID); assertThat(response.id()).isEqualTo(MOCK_ID);
assertThat(response.getExtra("key")).contains("value"); assertThat(response.getExtra("key")).contains("value");
@@ -165,43 +165,124 @@ services:
restart: unless-stopped restart: unless-stopped
``` ```
Note that I'm using my own locally hosted PostgreSQL instances for the main and test database. Just add databases via SQL or PgAdmin and install the pgvector extension to each database manually. There is an offical ready-made pgvector docker image but if you already host a PostGreSQL database you need to add the extension yourself. Note that I'm using my own locally hosted PostgreSQL instances for the main and test database. Just add databases via SQL or PgAdmin and install the pgvector extension to each database manually. There is an official ready-made pgvector docker image but if you already host a PostgreSQL database you need to add the extension yourself.
Check the name of your PostGreSQL container: ## Installing pgvector on a self-hosted PostgreSQL container
```
docker ps pgvector is a PostgreSQL extension that adds a `vector` data type. It does not create a new database — it adds a `vector_store` table to your existing database. The shared library (`vector.so`) must be installed on the PostgreSQL server itself.
**Do not install it manually inside a running container.** Manual installations do not survive container recreation (e.g. after a `docker compose up --force-recreate` or image update). Instead, bake it into a custom Docker image.
### Step 1: Create a custom Dockerfile for PostgreSQL
Create a `Dockerfile.postgres` next to your `docker-compose.yaml`:
```dockerfile
FROM postgres:18.4
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates git build-essential postgresql-server-dev-18 \
&& git clone --branch v0.8.3 https://github.com/pgvector/pgvector.git \
&& cd pgvector && make && make install \
&& cd .. && rm -rf pgvector \
&& apt-get purge -y ca-certificates git build-essential postgresql-server-dev-18 \
&& apt-get autoremove -y \
&& rm -rf /var/lib/apt/lists/*
``` ```
Enter your container via bash: - `FROM postgres:18.4` — this is still the standard postgres image, not the pgvector image
- `ca-certificates` — required for git to verify GitHub's SSL certificate during build
- `postgresql-server-dev-18` — provides the PostgreSQL header files needed to compile pgvector
``` ### Step 2: Update docker-compose.yaml to use the custom image
docker exec -it 876fb382969f bash
```
Before working on your database backup your databases:
```
su - postgres -c "pg_dumpall > /tmp/backup200526.sql"
#exit the container and copy the backup file to local file system
docker cp 876fb382969f:/tmp/backup200526.sql . ```yaml
services:
db:
container_name: postgres
labels:
- "com.centurylinklabs.watchtower.enable=false"
build:
context: .
dockerfile: Dockerfile.postgres
network: host
restart: always
environment:
POSTGRES_USER: ${DB_USER}
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_DB: ${DB_NAME}
networks:
- pg_network
volumes:
- /home/pi/docker/postgresql:/var/lib/postgresql
ports:
- "5432:5432"
networks:
pg_network:
external: true
``` ```
Install dependencies, build and install pgvector: ### Step 3: Build the image and recreate the container
apt-get update
apt-get install -y build-essential git postgresql-server-dev-all
```
git clone https://github.com/pgvector/pgvector.git
cd pgvector
make
make install
docker restart 876fb382969f
```
Enter PostGreSQL container and create pgvector extension for each databse:
```
docker exec -it <container-name> psql -h localhost -U <db-user> -d <db-name>
CREATE EXTENSION vector; Before recreating, back up your databases:
```bash
docker exec -it <container-name> bash
su - postgres -c "pg_dumpall > /tmp/backup.sql"
exit
docker cp <container-name>:/tmp/backup.sql .
``` ```
Build the image. Use `--network=host` to ensure the build container can reach GitHub:
```bash
docker build --network=host -t postgres-pgvector -f Dockerfile.postgres .
```
Recreate the container
```bash
docker compose up -d --force-recreate
```
### Step 4: Enable the extensions in each database
Run this once per database that needs pgvector:
```bash
docker exec -it <container-name> psql -U <db-user> -d <db-name>
```
```sql
CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
```
`uuid-ossp` is also required — Spring AI's `vector_store` table uses `uuid_generate_v4()` for its primary key.
### Step 5: Add pgvector config to application.yaml
Spring AI's pgvector auto-configuration requires these properties:
```yaml
spring:
ai:
vectorstore:
pgvector:
dimensions: 1536 # must match your embedding model output size
distance-type: COSINE_DISTANCE
index-type: HNSW
initialize-schema: true # auto-creates the vector_store table on startup
```
`dimensions` depends on the embedding model:
- `text-embedding-ada-002` / `text-embedding-3-small` → 1536
- `text-embedding-3-large` → 3072
`initialize-schema: true` means Spring AI will create the `vector_store` table automatically on first startup — no manual SQL needed.
# Appendix: Additional config for developing in Code-Server # Appendix: Additional config for developing in Code-Server
When using the code-server container there are additional config steps to mind: When using the code-server container there are additional config steps to mind:
+6 -2
View File
@@ -1,8 +1,12 @@
import { apiFetch } from "./client"; import { apiFetch } from "./client";
import type { PagedSearchResponse, SearchRequest, SearchResponse } from "../types/search"; import type {
PagedSearchResponse,
SearchRequest,
ServiceItem,
} from "../types/search";
export const search = (req: SearchRequest) => export const search = (req: SearchRequest) =>
apiFetch<PagedSearchResponse<SearchResponse>>("/search", { apiFetch<PagedSearchResponse<ServiceItem>>("/search", {
method: "POST", method: "POST",
body: JSON.stringify(req), body: JSON.stringify(req),
}); });
@@ -1,6 +1,6 @@
import { useEffect, useRef, useState, type SyntheticEvent } from 'react' import { useEffect, useRef, useState, type SyntheticEvent } from 'react'
import '../ui/Modal.scss' import '../ui/Modal.scss'
import { type PagedSearchResponse, type SearchResponse, type SearchRequest } from '../../types/search' import { type PagedSearchResponse, type ServiceItem, type SearchRequest } from '../../types/search'
import { search } from '../../api/searches' import { search } from '../../api/searches'
import type { ServiceType } from '../../types/connection' import type { ServiceType } from '../../types/connection'
@@ -14,11 +14,13 @@ interface Props {
export function SearchModal({ serviceType, label, appUrl, username, onClose }: Readonly<Props>) { export function SearchModal({ serviceType, label, appUrl, username, onClose }: Readonly<Props>) {
const [query, setQuery] = useState('') const [query, setQuery] = useState('')
const [results, setResults] = useState<PagedSearchResponse<SearchResponse> | null>(null) const [results, setResults] = useState<PagedSearchResponse<ServiceItem> | null>(null)
const [loading, setLoading] = useState(false) const [loading, setLoading] = useState(false)
const [error, setError] = useState<string | null>(null) const [error, setError] = useState<string | null>(null)
//TODO: implement aiSearch
const [aiSearch, setAiSearch] = useState(false);
const firstInputRef = useRef<HTMLInputElement>(null) const firstInputRef = useRef<HTMLInputElement>(null)
const dialogRef = useRef<HTMLDialogElement>(null) const dialogRef = useRef<HTMLDialogElement>(null)
useEffect(() => { useEffect(() => {
@@ -41,7 +43,7 @@ export function SearchModal({ serviceType, label, appUrl, username, onClose }: R
setError(null) setError(null)
setLoading(true) setLoading(true)
try { try {
const req: SearchRequest = { appUrl, serviceType, username, query} const req: SearchRequest = { appUrl, serviceType, username, query, aiSearch}
const res = await search(req) const res = await search(req)
console.log(req) console.log(req)
setResults(res) setResults(res)
+17 -16
View File
@@ -1,25 +1,26 @@
import type { ServiceType } from "./connection"; import type { ServiceType } from "./connection";
export interface SearchRequest { export interface SearchRequest {
appUrl: string appUrl: string;
serviceType: ServiceType serviceType: ServiceType;
username: string username: string;
query: string | null query: string | null;
aiSearch: boolean;
} }
export interface SearchResponse { export interface ServiceItem {
id: string id: string;
title: string title: string;
description: string | null description: string | null;
extraData: Record<string, unknown> extraData: Record<string, unknown>;
} }
export interface PagedSearchResponse<T> { export interface PagedSearchResponse<T> {
content: T[] content: T[];
page: number page: number;
pageSize: number pageSize: number;
totalElements: number totalElements: number;
first: boolean first: boolean;
last: boolean last: boolean;
sort: string sort: string;
} }