Spring-ai Feature Request: Custom Fields for Vector Database Filtering in Spring AI

Expected Behavior

The VectorStore interface and related classes in Spring AI should support custom fields for vector database filtering. This would allow users to:

1. Add custom fields to documents
2. Perform similarity searches with filtering based on these custom fields

Example code for how this might look:

// Adding a document with custom fields
Document doc = Document.builder()
    .withContent("Sample content")
    .withCustomField("category", "technology")
    .withCustomField("publishDate", LocalDate.now())
    .build();

vectorStore.add(List.of(doc));

// Performing a search with custom field filtering
SearchRequest request = SearchRequest.builder()
    .withQuery("AI advancements")
    .withCustomFieldFilter("category", "technology")
    .withCustomFieldFilter("publishDate", LocalDate.now().minusDays(7))
    .build();

List<Document> results = vectorStore.similaritySearch(request);

Current Behavior

Currently, the VectorStore interface does not provide a way to add custom fields and limited to the Document class. The Document class has a fixed set of fields. This limits the ability to perform more granular and domain-specific searches.

Context

This feature is needed because:

1. Many applications require domain-specific metadata for documents (e.g., publication date, author, category).
2. Users often need to filter search results based on these custom attributes in addition to similarity matching.

Comment From: dafriz

Passing a Filter.Expression to the SearchRequest covers this using metadata. For example:

// Adding a document with metadata fields
Document doc = Document.builder()
    .withContent("Sample content")
    .withMetadata("category", "technology")
    .build();

vectorStore.add(List.of(doc));

// Performing a search with filter expression
SearchRequest request = SearchRequest
    .query("AI advancements")
    .withFilterExpression(
        new FilterExpressionBuilder()
             .eq("category", "technology")
             .build()   
    );

List<Document> results = vectorStore.similaritySearch(request);