This PR adds supports for Elasticsearch vector search.
To achieve compatibility with both Elasticsearch versions 7 and 8, I implemented it using Elasticsearch's Java Low Level REST Client.
To-do: - [x] Add ElasticsearchAiSearchFilterExpressionConverter - [x] Add support for FilterExpression - [x] Add support for various vector functions provided by Elasticsearch
Comment From: JM-Lab
Just finished up the Elasticsearch Vector Store support. Next, I'll be working on adding support for OpenSearch (https://opensearch.org/) vector store. Additionally, breaking down tasks for separate PR:
- [ ] Add autoconfiguration
- [ ] Write README
Let me know your thoughts.
Comment From: markpollack
This is a great contribution. I believe @tzolov has started to review. I'll put it into 0.8.0 as a stretch goal for now.
Comment From: JM-Lab
If we use elasticsearch-java, we might need to either split Vectorstore for Elasticsearch 7 and 8, or let users add elasticsearch-java version matching their elasticsearch server version. What's your preference? I'd love to hear your thoughts on this.
Comment From: markpollack
I think moving to use the latest version that focuses on elastic search 8 and to build upon the elasticsearch-java will be the best use of everyone's time. Sorry for the delay in responding, but we are still very interested in this contribution and appreciate your interest/help.
Comment From: JM-Lab
@markpollack I understand, and I'll update to support the latest version, Elasticsearch 8, and commit soon.
Comment From: JM-Lab
@markpollack @tzolov Review code with Elasticsearch 8 support using elasticsearch-java library as discussed.
Comment From: tzolov
Hi @JM-Lab , Finally i had some time today to start reviewing the PR. After rebasing it on top of upstream main there were some compilation issues i had to fix. Also i had to add the project the parent POM and to the BOM pom.
After this the filter expression test passes but the IT fails for half of the tests. Here is my review branch: https://github.com/tzolov/spring-ai/pull/new/gh-234-pr
Please have a look, hopefully you would be able to resolve those.
Comment From: JM-Lab
Hi @tzolov
I've reviewed the PR. I cherry-picked your commit 'Rebase, fix compilation and style errors, add missing dependencies in parent pom and Bom pom' and resolved the issues with Elasticsearch8VectorStoreIT. Additionally, I renamed it to ElasticsearchVectorStoreIT.
Please review the changes. Looking forward to your feedback!
Comment From: tzolov
Thanks @JM-Lab , 75 commits and 656 files changed, doesn't seem right. Can you please squash the commits and make sure that only the ElasticSearch related files a touched. It will be even better if you copy the ElasticSearch related files into a fresh new PR on top of the upstream main.
Comment From: JM-Lab
@tzolov
I've just created a new PR focusing solely on the ElasticSearch files, on the main branch. You can find it here: https://github.com/spring-projects/spring-ai/pull/471.
Going to close this current one. Thanks!
Comment From: tzolov
Thanks @JM-Lab I will have a look