I'm running a Spring Boot API in a container on Heroku, and I think there might be a performance issue. In the image above, you can see that towards the right of the graph (provided by Heroku) there is a large uptick in memory usage. This increase correlates with the time I deployed my API with 2.5 - before this I was using 2.4.5. At the right most edge of the graph you can see that there is a steep drop to normal levels. Again, this correlates with a deployment, this time reverting 2.5 back to 2.4.5. The blue squares on the top graph of type activity are deployments
Additionally, I noticed that startup times for 2.5 were slower, around ~20-25 seconds for the 2.5 version of my api, and ~12-16 seconds for my 2.4.5 version.
Nothing else has changed in my applications, I've just bumped from 2.4.5 up to 2.5, and then back down again.
I'm not sure what this could be related to. A likely culprit in my mind would be the datasource changes recently introduced. I can't really see anything else in the release notes that might have this effect.
For now I'm remaining back on 2.4.5 - I'm not really sure how to further investigate this issue but wanted to say what a huge fan of Spring I am.
Cheers
Alex
PS In case it helps, here's my dependencies and Dockerfile:
build.gradle:
Plugins
plugins {
id "java"
// Changing from 2.4.5 to 2.5 below appeared to cause the performance issues shown in the graph above
id "org.springframework.boot" version "2.4.5"
id 'io.spring.dependency-management' version '1.0.11.RELEASE'
id "jacoco"
id "org.flywaydb.flyway" version "7.7.3"
id "com.github.ben-manes.versions" version "0.38.0"
}
Dependencies
dependencies {
// Spring Boot
implementation('org.springframework.boot:spring-boot-starter-actuator')
implementation('org.springframework.boot:spring-boot-starter-data-jpa')
implementation('org.springframework.boot:spring-boot-starter-hateoas')
implementation('org.springframework.boot:spring-boot-starter-web')
implementation('org.springframework.boot:spring-boot-starter-security')
implementation('org.springframework.boot:spring-boot-starter-aop')
implementation('org.springframework.boot:spring-boot-starter-validation')
runtimeOnly('org.springframework.boot:spring-boot-devtools')
annotationProcessor "org.springframework.boot:spring-boot-configuration-processor"
// Testing
testImplementation(platform('org.junit:junit-bom:5.7.0'))
testImplementation('org.junit.jupiter:junit-jupiter')
testImplementation('org.springframework.boot:spring-boot-starter-test')
testImplementation("org.assertj:assertj-core:3.19.0")
testImplementation 'org.mockito:mockito-core:3.10.0'
testImplementation "org.springframework.security:spring-security-test"
testImplementation 'com.intuit.karate:karate-junit5:1.0.1'
testImplementation "net.masterthought:cucumber-reporting:5.5.3"
// Persistence
implementation('com.h2database:h2')
implementation('org.postgresql:postgresql')
implementation('org.springframework.boot:spring-boot-starter-jdbc')
implementation "org.flywaydb:flyway-core:7.7.3"
// Swagger
implementation 'org.springdoc:springdoc-openapi-ui:1.5.8'
implementation 'org.springdoc:springdoc-openapi-hateoas:1.5.8'
implementation 'org.springdoc:springdoc-openapi-security:1.5.8'
// Observability
implementation 'io.sentry:sentry-spring-boot-starter:4.3.0'
implementation 'io.sentry:sentry-logback:4.3.0'
implementation 'io.honeycomb.beeline:beeline-spring-boot-starter:1.5.1'
implementation 'nl.basjes.parse.useragent:yauaa:5.23'
// Security
implementation 'com.auth0:auth0-spring-security-api:1.4.1'
implementation 'org.springframework.security:spring-security-oauth2-resource-server'
implementation 'org.springframework.security:spring-security-oauth2-jose'
implementation 'org.springframework.security:spring-security-config'
// Outgoing Email
implementation 'com.sendgrid:sendgrid-java:4.7.2'
// Serialisation
implementation 'com.fasterxml.jackson.core:jackson-databind:2.11.1'
}
Dockerfile:
FROM gradle:7.0.2-jdk16 as build-stage
WORKDIR /build
COPY src /build/src
COPY gradlew /build/gradlew
COPY gradle /build/gradle
COPY build.gradle /build/build.gradle
COPY settings.gradle /build/settings.gradle
RUN /build/gradlew clean build -x test
# Running container build begins
FROM adoptopenjdk:16-jre-openj9-focal
# copy application JAR
COPY --from=build-stage build/build/libs/api.jar /api.jar
# specify default command
CMD ["sh", "-c", " java ${JAVA_OPTS} -Xverify:none -jar /api.jar"]
Comment From: bclozel
Hello @AlexanderNZ
Sorry you ran into this problem while upgrading. The information you’ve provided so far might be not enough for us to pinpoint and fix the problem.
Could you get the following for both 2.4.5 and 2.5.0 so we can compare?
- application startup tracking information (you can retrieve it as JSON with an actuator endpoint)
- heap dump or any memory profiler report that show dominator objects
The issue you’re seeing right now might be related to a dependency upgrade or something hard to track down.
thanks!
Comment From: AlexanderNZ
Hi @bclozel , Here are the results form the startup actuator endpoint for 2.4.5 and 2.5.0 :) Working on a heap dump for both presently. Github didn't let me upload JSON so have appended .txt to the end of the files.
2-5-0-startup.json.txt 2-4-5-startup.json.txt
Comment From: bclozel
I'm confused. Those startup dumps are quite similar: * they're both around 15 secs * 2.5.0 has less beans declared overall
It seems that the vast majority of startup time is being taken by:
1. the Honeycomb BeelineAutoConfig (2200ms)
2. Flyway migrations (1500ms)
3. the JwtDecoder bean creation (960ms)
4. quite a few repositories (approx. 2000ms)
5. the resource handling infrastructure seems a bit slow here, is it being instrumented with a dependency or library you're using here?
Am I missing something here?
Comment From: AlexanderNZ
That is confusing. I didn't manage to get a heap dump - I'm currently packaging this application in a container with Open J9, not Hotspot and I wonder if that might be it? I've just changed the container to Hotspot and start time has dropped to nine seconds - which is a bit confusing.
Or maybe it's just a weird quirk of Heroku running the container, maybe some auto-added config as part of that?
If you're not getting loads of reports like this I'd say it's probably something that I'm doing and NOT something related to Spring, even though it's only apparent when I bump to 2.5.
I'll do some more investigation and let you know if I have any discoveries, but thanks for investigating for me!
Comment From: bclozel
Thanks for getting back to us @AlexanderNZ . I'm not familiar with specific runtime differences between JVMs, maybe they don't manage the available memory the same way and put more/less pressure on the GC during startup?
Let us know if you find something interesting!