This PR provides the support of Oracle Database 23ai as a VectorStore.
Comment From: loiclefevre
@eddumelendez, is there anything to do remaining on my side?
Comment From: tzolov
Hey @loiclefevre , great contribution! Thank you! I've started reviewing it but I'm unable to run the integration test OracleVectorStoreTests.
Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '.*DATABASE IS READY TO USE!.*\s'
at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:47)
LogMessageWaitStrategy.java:47
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:52)
AbstractWaitStrategy.java:52
at org.testcontainers.oracle.OracleContainer.waitUntilContainerStarted(OracleContainer.java:81)
OracleContainer.java:81
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:500)
GenericContainer.java:500
... 62 common frames omitted
15:55:02.101 [main] ERROR tc.gvenzl/oracle-free:23-slim -- Log output from the failed container:
CONTAINER: starting up...
CONTAINER: first database startup, initializing...
CONTAINER: uncompressing database data files, please wait...
CONTAINER: done uncompressing database data files, duration: 2 seconds.
CONTAINER: starting up Oracle Database...
LSNRCTL for Linux: Version 23.0.0.0.0 - Production on 13-JUN-2024 13:54:05
Copyright (c) 1991, 2024, Oracle. All rights reserved.
Starting /opt/oracle/product/23ai/dbhomeFree/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 23.0.0.0.0 - Production
System parameter file is /opt/oracle/product/23ai/dbhomeFree/network/admin/listener.ora
Log messages written to /opt/oracle/diag/tnslsnr/6a7153eb4422/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC_FOR_FREE)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC_FOR_FREE)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Any suggestions?
Btw the docker's log shows this:
2024/06/13 13:54:01 Pinging Docker...
2024/06/13 13:54:01 Docker daemon is available!
2024/06/13 13:54:01 Starting on port 8080...
2024/06/13 13:54:01 Started!
2024/06/13 13:54:01 New client connected: 172.17.0.1:59682
2024/06/13 13:54:01 Adding {"label":{"org.testcontainers.lang=java":true,"org.testcontainers.sessionId=76881e23-5db2-4694-99f7-e12e5f25fe61":true,"org.testcontainers.version=1.19.8":true,"org.testcontainers=true":true}}
2024/06/13 13:55:02 Signal received
2024/06/13 13:55:02 EOF
2024/06/13 13:55:02 Removed 1 container(s), 0 network(s), 0 volume(s) 0 image(s)
@eddumelendez perhaps you can have some insides here as well?
Comment From: loiclefevre
Thanks @tzolov.
Timeout. Was this the first time the image was pulled? If so, could you please retry if you have the image locally? It may appear that I need to increase the Testcontainers startup timeout using withStartupTimeoutSeconds().
Comment From: tzolov
@loiclefevre i tried couple of times already and the images is listed locally.
Comment From: loiclefevre
@loiclefevre i tried couple of times already and the images is listed locally.
Do you have anything inside your ~/.testcontainers.properties?
Comment From: loiclefevre
I just ran the tests:
[INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 262.0 s -- in org.springframework.ai.vectorstore.OracleVectorStoreTests
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:33 min
[INFO] Finished at: 2024-06-13T16:46:49+02:00
[INFO] ------------------------------------------------------------------------
Comment From: loiclefevre
Following the logs from the container:
CONTAINER: starting up...
CONTAINER: first database startup, initializing...
CONTAINER: uncompressing database data files, please wait...
CONTAINER: done uncompressing database data files, duration: 6 seconds.
CONTAINER: starting up Oracle Database...
LSNRCTL for Linux: Version 23.0.0.0.0 - Production on 13-JUN-2024 14:48:08
Copyright (c) 1991, 2024, Oracle. All rights reserved.
Starting /opt/oracle/product/23ai/dbhomeFree/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 23.0.0.0.0 - Production
System parameter file is /opt/oracle/product/23ai/dbhomeFree/network/admin/listener.ora
Log messages written to /opt/oracle/diag/tnslsnr/975a502b723c/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC_FOR_FREE)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC_FOR_FREE)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 23.0.0.0.0 - Production
Start Date 13-JUN-2024 14:48:09
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Default Service FREE
Listener Parameter File /opt/oracle/product/23ai/dbhomeFree/network/admin/listener.ora
Listener Log File /opt/oracle/diag/tnslsnr/975a502b723c/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC_FOR_FREE)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
The listener supports no services
The command completed successfully
ORACLE instance started.
Total System Global Area 1603726344 bytes
Fixed Size 5360648 bytes
Variable Size 788529152 bytes
Database Buffers 805306368 bytes
Redo Buffers 4530176 bytes
Database mounted.
Database opened.
CONTAINER: Resetting SYS and SYSTEM passwords.
User altered.
User altered.
CONTAINER: Creating app user for default pluggable database.
Session altered.
User created.
Grant succeeded.
CONTAINER: DONE: Creating app user for default pluggable database.
CONTAINER: Executing user-defined scripts...
CONTAINER: running /container-entrypoint-initdb.d/initialize.sql ...
System altered.
ORACLE instance shut down.
ORACLE instance started.
Total System Global Area 1603726344 bytes
Fixed Size 5360648 bytes
Variable Size 335544320 bytes
Database Buffers 184549376 bytes
Redo Buffers 4530176 bytes
Vector Memory Area 1073741824 bytes
Database mounted.
Database opened.
CONTAINER: DONE: running /container-entrypoint-initdb.d/initialize.sql
CONTAINER: DONE: Executing user-defined scripts.
#########################
DATABASE IS READY TO USE!
#########################
####################################################################
CONTAINER: The following output is now from the alert_FREE.log file:
####################################################################
...
Comment From: loiclefevre
@tzolov, your containers logs seem to stop at:
TNSLSNR for Linux: Version 23.0.0.0.0 - Production
System parameter file is /opt/oracle/product/23ai/dbhomeFree/network/admin/listener.ora
Log messages written to /opt/oracle/diag/tnslsnr/6a7153eb4422/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC_FOR_FREE)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC_FOR_FREE)))
STATUS of the LISTENER
------------------------
Alias LISTENER
You have nothing after?
Comment From: tzolov
No nothing more in the logs
Comment From: loiclefevre
What is your environment? Operating system, container engine?
Comment From: gvenzl
It appears that the container itself died:
2024/06/13 13:54:01 New client connected: 172.17.0.1:59682
2024/06/13 13:54:01 Adding {"label":{"org.testcontainers.lang=java":true,"org.testcontainers.sessionId=76881e23-5db2-4694-99f7-e12e5f25fe61":true,"org.testcontainers.version=1.19.8":true,"org.testcontainers=true":true}}
2024/06/13 13:55:02 Signal received
2024/06/13 13:55:02 EOF
2024/06/13 13:55:02 Removed 1 container(s), 0 network(s), 0 volume(s) 0 image(s)
Note especially the Signal received and EOF.
@eddumelendez, would you know the best way to determine the container state and why the runtime killed it?
@tzolov, your environment details would indeed be helpful, also whether or not you are running on an ARM chip or x86.
Thanks,
Comment From: eddumelendez
Hi, when running testcontainers, the operating system should be logged. IIRC, @loiclefevre is using colima, probably using emulation for x86 images and @tzolov could be using Docker on M1 where issues have been reported when using Oracle images. Testcontainers Cloud can help here to execute OracleFree containers 😉
Comment From: tzolov
Hi there,
Indeed switched last week to Christian’s MacBook Pro M3 Max and indeed there is a log warning when running the test:
22:23:49.350 [main] WARN tc.gvenzl/oracle-free:23-slim -- The architecture 'amd64' for image 'gvenzl/oracle-free:23-slim' (ID sha256:e315f0be9ea21c02210541b421fa7918393ed9acc99d9cc0b73e48a3f14fd246) does not match the Docker server architecture 'arm64'. This will cause the container to execute much more slowly due to emulation and may lead to timeout failures.
Comment From: tzolov
I could not resolve the issue by extending the timeouts but i've successfully run the tests on another (Intel based) machine.
Comment From: loiclefevre
... IIRC, @loiclefevre is using colima...
Not at all, I'm using WSL2 on Windows. No MAC in my house, never 🤣
Comment From: loiclefevre
I could not resolve the issue by extending the timeouts but i've successfully run the tests on another (Intel based) machine.
Indeed, increasing the timeout will not help here. MAC is the culprit 😈
Comment From: tzolov
@loiclefevre I've rebased, squashed and merged the PR with some minor javadoc adjustments and also renamed some class names to align them with the common store naming convention.
Then I decided to adding the missing auto-configuration tests, in a separate PR/commit, and found an awkward issue.
While the oracle jar dependencies in the oracle-vector-store project are expectedly resolved to23.4.0.24.05, when you check the oracle dependencies in the auto-configuration and boot starter projects they are downgraded to 21.9.0.0! Not what causes the downgrade?
./mvnw dependency:tree -pl vector-stores/spring-ai-oracle-store
./mvnw dependency:tree -pl spring-ai-spring-boot-autoconfigure
./mvnw dependency:tree -pl spring-ai-spring-boot-starters/spring-ai-starter-oracle-store
Can you please confirm this?
I will open another Issue and PR for the auto-config testing (and dependency management)
Comment From: loiclefevre
Yes, confirmed and strange.
However:
...
[INFO] +- org.springframework.boot:spring-boot-starter:jar:3.3.0:compile
[INFO] | +- org.springframework.boot:spring-boot:jar:3.3.0:compile
[INFO] | | \- org.springframework:spring-context:jar:6.1.8:compile
[INFO] | +- org.springframework.boot:spring-boot-autoconfigure:jar:3.3.0:compile
[INFO] | +- org.springframework.boot:spring-boot-starter-logging:jar:3.3.0:compile
...
And inside .m2\repository\org\springframework\boot\spring-boot-dependencies\3.3.0\spring-boot-dependencies-3.3.0.pom we can find:
<oracle-database.version>21.9.0.0</oracle-database.version>
Comment From: loiclefevre
@tzolov, probably this must be reviewed:
https://github.com/spring-projects/spring-boot/blob/main/spring-boot-project/spring-boot-dependencies/build.gradle
Comment From: tzolov
@loiclefevre you can rise an issue agains the boot repo if you think the version there needs an update.
This won't change the fact that we have to find a work around.
Playing with few options right now
Comment From: tzolov
The 21.9.0.0 leads to ClassNoDef errors.
Comment From: aalmiray
Temporary workaround is to overwrite the value of the managed property (from spring-dependencies) that defines the oracle version to be used.
Comment From: tzolov
Indeed this is what I did in: https://github.com/spring-projects/spring-ai/pull/867
@loiclefevre , @aalmiray feel free to review it. I've forced the oracle dependencies in the auto-conf and boot POMs. Added a simple Auto-conf test that passes after the fix. I also renamed few class names to align them with the reset of the project.
Comment From: tzolov
Rebased, squashed and merged at a5d92ec30f46c9e476078ab0c5681aadee761a8f
Related follow up changes in PR: https://github.com/spring-projects/spring-ai/pull/867
Comment From: loiclefevre
Thanks a lot @tzolov 🙏