Our Spring Boot application is running in AWS Elastic Container Service in production and connects to an AWS RDS Postgresql 10.7 engine in the RDS serverless db service.
Our maven project is simple, it uses spring-boot-starter-parent as maven parent. Also we have spring-boot-starter-data-jpa and finally org.postgresql:postgresql as a runtime dependency.
With spring boot 2.3.3 that setup works out of the box and connects to RDS without any issues
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.3.3.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
...
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>runtime</scope>
</dependency>
If we update to using Spring Boot 2.3.4 all database tests we run locally against Postgresql 10.7 docker images still work, but the application consistently fail in ECS to connect to the RDS Postgresql database.
After doing a diff of the maven dependency:tree output for the two versions I noticed the postgresql driver had been updated from 42.2.14 in Spring Boot 2.3.3 to 42.2.16 in Spring Boot 2.3.4.
If I override the postgresql.version to 42.2.14 in our pom.xml our application connects nicely to the RDS Postgresql database again.
This is working for us in production with Spring Boot 2.3.4
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.3.4.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<properties>
<java.version>11</java.version>
<!-- Override Spring Boot postgresql version to be able to connect to AWS RDS Serverless Postgres in ECS -->
<postgresql.version>42.2.14</postgresql.version>
</properties>
<dependencies>
...
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>runtime</scope>
</dependency>
This is probably an issue in the postgresql driver 42.2.16 - but I figured I should post the report and workaround here too since the spring-boot-dependencies project have a depency property referring the 42.2.16 version specifically.
Comment From: cawrites
This one sounds interesting. Are you going to implement the workaround?
Comment From: wilkinsona
Thanks for letting us know. This is probably a duplicate of https://github.com/spring-projects/spring-boot/issues/23491. Please raise this with the Postgres team if you haven't done so already. We'll pick up any new Postrges release in due course. For the record, we won't downgrade as that would affect everyone using Spring Boot and Postgres and the problem appears to be more isolated than that.
Comment From: frjtrifork
Thanks for letting us know. This is probably a duplicate of #23491. Please raise this with the Postgres team if you haven't done so already. We'll pick up any new Postrges release in due course. For the record, we won't downgrade as that would affect everyone using Spring Boot and Postgres and the problem appears to be more isolated than that.
You are welcome - and yes that other report look identical to our issue.
I will file a bug report with postgresql about this problem as well. Update: I have filed the issue here: https://www.postgresql.org/message-id/16641-e8ea5e56aa47bd30%40postgresql.org
For the record the stack trace we get with 42.2.16 when connecting to the RDS instance is this (we get the same error with 42.2.15 as well):
Caused by: org.flywaydb.core.internal.exception.FlywaySqlException:
--
| Unable to obtain connection from database: The connection attempt failed.
| -------------------------------------------------------------------------
| SQL State : 08001
| Error Code : 0
| Message : The connection attempt failed.
| at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:65) ~[flyway-core-6.4.4.jar:na]
| at org.flywaydb.core.internal.jdbc.JdbcConnectionFactory.<init>(JdbcConnectionFactory.java:80) ~[flyway-core-6.4.4.jar:na]
| at org.flywaydb.core.Flyway.execute(Flyway.java:456) ~[flyway-core-6.4.4.jar:na]
| at org.flywaydb.core.Flyway.migrate(Flyway.java:159) ~[flyway-core-6.4.4.jar:na]
| at org.springframework.boot.autoconfigure.flyway.FlywayMigrationInitializer.afterPropertiesSet(FlywayMigrationInitializer.java:65) ~[spring-boot-autoconfigure-2.3.4.RELEASE.jar:2.3.4.RELEASE]
| at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1853) ~[spring-beans-5.2.9.RELEASE.jar:5.2.9.RELEASE]
| at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1790) ~[spring-beans-5.2.9.RELEASE.jar:5.2.9.RELEASE]
| ... 27 common frames omitted
| Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
| at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:315) ~[postgresql-42.2.16.jar:42.2.16]
| at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[postgresql-42.2.16.jar:42.2.16]
| at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225) ~[postgresql-42.2.16.jar:42.2.16]
| at org.postgresql.Driver.makeConnection(Driver.java:465) ~[postgresql-42.2.16.jar:42.2.16]
| at org.postgresql.Driver.connect(Driver.java:264) ~[postgresql-42.2.16.jar:42.2.16]
| at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) ~[HikariCP-3.4.5.jar:na]
| at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:358) ~[HikariCP-3.4.5.jar:na]
| at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206) ~[HikariCP-3.4.5.jar:na]
| at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:477) ~[HikariCP-3.4.5.jar:na]
| at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:560) ~[HikariCP-3.4.5.jar:na]
| at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115) ~[HikariCP-3.4.5.jar:na]
| at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112) ~[HikariCP-3.4.5.jar:na]
| at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:56) ~[flyway-core-6.4.4.jar:na]
| ... 33 common frames omitted
| Caused by: java.io.EOFException: null
| at org.postgresql.core.PGStream.receiveChar(PGStream.java:443) ~[postgresql-42.2.16.jar:42.2.16]
| at org.postgresql.core.v3.ConnectionFactoryImpl.enableGSSEncrypted(ConnectionFactoryImpl.java:436) ~[postgresql-42.2.16.jar:42.2.16]
| at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:144) ~[postgresql-42.2.16.jar:42.2.16]
| at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213) ~[postgresql-42.2.16.jar:42.2.16]
| ... 45 common frames omitted
Comment From: frjtrifork
Got a reply from Postgresql bug list - and confirmed that the suggested workaround works with 42.2.16 in RDS.
"Well the problem is that the driver attempts to make a GSS encrypted connection and AWS doesn't adhere to the PostgreSQL protocol.
You can add gssEncMode=disable to the connection and the driver will not
attempt the encrypted connection.
We will be releasing shortly a version which does not default to attempting
the gssEncMode.
You really should be filing this but with Amazon as their product does not
adhere to the PostgreSQL protocol
Dave"
I filed the issue at the AWS RDS team as suggested as well - but that issue is private to my account. I will update this thread if they say they will fix it.
But good to know Postgresql will release a driver that does not default to using the GSS encryption feature.
(Note that the changelog here https://jdbc.postgresql.org/ says that after 42.2.15 GSS connect attempt should attempt GSSAPI first, then fall back to SSL, and finally plain text connection - but that did not happen in our case in RDS - we had to add the flag to disable gssEncMode)
Comment From: wilkinsona
@frjtrifork Thanks for sharing what you heard from the Postgres team.