It's tested by tryingToUseJaninoWhenItIsNotOnTheClasspathFailsGracefully in LoggingApplicationListenerTests at the moment. The test relies on Logback's behavior when an attempt is made to use Janino when it isn't on the classpath. That behavior is an implementation detail of Logback and it's changed in 1.5.16. The test just needs a logging system that throws a Throwable from initialize so we should provide one so that we have complete control over its behavior.