I recently encountered a production issue as a consequence of the following problem: 1. A Spring Boot application with Liquibase was updated from version 1 to version 2 on Kubernetes. 2. The application started up and applied a new Liquibase changeset. 3. The startup failed for different reason and an automatic rollback was applied. 4. Version 1 of the application came back up, seeing the change set of version 2. 5. The application started but misbehaved in a rather distructive manner due to an unforseen consequence of the new data model in combination with the old SQL.
For this reason, we wanted to secure a way to avoid that an application can start up with "future changesets" and rather allow such behaviour by configuration. I was wondering if this ability would be of interest to be added to Spring Boot?
We solved this as follows:
@Bean
@ConditionalOnClass(SpringLiquibase.class)
public ApplicationListener<ApplicationStartedEvent> liquibaseChangeLogComparator(
ApplicationContext applicationContext,
@Autowired(required = false) DataSource dataSource,
@Autowired(required = false) List<SpringLiquibase> springLiquibases
) {
if (dataSource == null || springLiquibases == null) {
return event -> LOGGER.debug("Liquibase is not enabled and change logs are not compared");
}
MethodHandle handle;
try {
Method method = SpringLiquibase.class.getDeclaredMethod("createLiquibase", Connection.class);
method.setAccessible(true);
handle = MethodHandles.lookup().unreflect(method);
} catch (Exception e) {
return event -> LOGGER.warn("Failed to resolve Liquibase constructor", e);
}
List<Pattern> known = properties.getChangeSets().entrySet().stream()
.filter(Map.Entry::getValue)
.map(entry -> Pattern.compile(entry.getKey()))
.toList();
return event -> {
Set<ChangeSetInfo> applied = new HashSet<>(), unexpected = new HashSet<>();
try (Connection conn = dataSource.getConnection()) {
for (SpringLiquibase springLiquibase : springLiquibases) {
boolean dropFirst = springLiquibase.isDropFirst();
springLiquibase.setDropFirst(false);
try (Liquibase liquibase = (Liquibase) handle.invoke(springLiquibase, conn)) {
liquibase.getDatabaseChangeLog().getChangeSets().stream()
.map(changeSet -> new ChangeSetInfo(changeSet.getId(), changeSet.getChangeLog().getFilePath(), changeSet.getAuthor()))
.forEach(applied::add);
UnexpectedChangesetsCommandStep.listUnexpectedChangeSets(liquibase.getDatabase(),
liquibase.getDatabaseChangeLog(),
new Contexts(springLiquibase.getContexts()),
new LabelExpression()).stream().map(changeSet -> new ChangeSetInfo(changeSet.getId(),
changeSet.getChangeLog(),
changeSet.getAuthor())).forEach(unexpected::add);
}
springLiquibase.setDropFirst(dropFirst);
}
} catch (Throwable t) {
throw new IllegalStateException("Failed to process Liquibase meta information");
}
unexpected.removeAll(applied);
if (unexpected.isEmpty()) {
LOGGER.debug("Did not find unaccounted change sets");
} else {
LOGGER.error("Found applied unaccounted change set {}", unexpected);
SpringApplication.exit(applicationContext);
}
};
}
Ideally, reflection would not be needed, but the SpringLiquibase bean is defined in the Liquibase dependency. If one changed the code there, one could also reduce the additional use of a data base connection.
Comment From: Shivam1-123
Is it for beginners?
Comment From: wilkinsona
No, I'm afraid not. The issue's labelled as for: team-meeting which means that we want to discuss it together to figure out what to do. If you're looking for an issue that's suitable for beginners, please keep an eye out for any that are labelled as ideal for contribution.
Comment From: wilkinsona
@raphw a couple of thoughts before we have a change to discuss this one as a team:
- Could you implement a
Customizer<Liquibase>bean to capture theLiquibaseinstance, rather than relying on reflection? The auto-configuredSpringLiquibasewill automatically have any such beans added to it. - Is an approach similar to that taken in Actuator's Liquibase endpoint applicable here?
Comment From: raphw
A Customizer as this type? We are not using Spring Security, and I cannot find any equivalent type on the class path.
As of today's Liquibase API, I cannot see that avoiding reflection is possible. The createLiquibase method is protected and can be overridden with custom implementations. The instance of Liquibase that is created in this method never escapes the method, so I would not understand how Spring can get hold of it.
Ideally, I would want to add a callback interface to SpringLiquibase which is then accepting the created instance once it is already created. This would also allow to interact with other API that is not currently exposed by SpringLiquibase such as registering a custom exec listener, so I would think this is a good idea, generally speaking.
Comment From: wilkinsona
Sorry, I should have fully-qualified the class name. It's liquibase.integration.spring.Customizer. Refreshing my memory a bit, it's new in Liquibase 4.28 and we added auto-configuration support in 3.4.0-M1. If you're using an earlier version of Boot, but happy to override the Liquibase version, I think you could use a bean post-processor to set the Customizer on the auto-configured SpringLiquibase bean.
Ideally, I would want to add a callback interface to SpringLiquibase which is then accepting the created instance once it is already created.
This is exact what Liquibase's Customizer does. SpringLiquibase calls it within createLiquibase.
Comment From: raphw
It would look like this then:
public class LiquibaseOrphanedChangeSetValidator implements Customizer<Liquibase>, ApplicationListener<ApplicationStartedEvent> {
private final Set<ChangeSetInfo> applied = new HashSet<>(), unexpected = new HashSet<>();
@Override
public void customize(Liquibase liquibase) {
try {
liquibase.getDatabaseChangeLog().getChangeSets().stream()
.map(changeSet -> new ChangeSetInfo(changeSet.getId(), changeSet.getChangeLog().getFilePath(), changeSet.getAuthor()))
.forEach(applied::add);
UnexpectedChangesetsCommandStep.listUnexpectedChangeSets(liquibase.getDatabase(),
liquibase.getDatabaseChangeLog(),
new Contexts(),
new LabelExpression()).stream().map(changeSet -> new ChangeSetInfo(changeSet.getId(),
changeSet.getChangeLog(),
changeSet.getAuthor())).forEach(unexpected::add);
} catch (LiquibaseException e) {
throw new IllegalStateException(e);
}
}
@Override
public void onApplicationEvent(ApplicationStartedEvent event) {
unexpected.removeAll(applied);
if (!unexpected.isEmpty()) {
SpringApplication.exit(event.getApplicationContext());
}
}
record ChangeSetInfo(String file, String id, String author) { }
}
I am not sure about the new Contexts() bit. Possibly one would need to resolve this from the SpringLiquibase before passing the customizer to a particular instance.
Comment From: philwebb
We discussed this today during our team call and we feel like the majority of this code belongs in Liquibase itself and not in Spring Boot. If there was a UnexpectedChangeSetsCustomizer class in Liquibase, the Spring Boot integration would be pretty light.
@raphw Perhaps you could start with a Liquibase contribution then we can add Spring Boot integration if that gets accepted?