I discovered an issue with Spring Boot Micrometer`s OpenTelemetry integration, version 3.2 and newer but older versions are probably also affected, in combination with the Google Cloud Logging Client, version 3.19.0 and newer. With this combination the logging MDC is no longer filled with tracing and baggage fields. I did an investigation and figured out the following:

  • The Google Cloud Logging client 3.19.0 now has support for OpenTelemetry. For every log event it will try to extract tracing information from the current OpenTelemetry Span to add to the meta data of the log event.
  • As logging is done before the OpenTelemetryAutoConfiguration initializes OpenTelemetry, parts of OpenTelemetry have already been initialized. The problematic parts that are initialized are the OpenTelemetry context storage classes like ContextStorage, LazyStorage and ContextStorageWrappers.
  • During Spring Boots initialization of OpenTelemetry, in the org.springframework.boot.actuate.autoconfigure.tracing.OpenTelemetryAutoConfiguration#otelCurrentTraceContext method, the EventPublishingContextWrapper instance is added to the OpenTelemetry context storage via the ContextStorage.addWrapper method.
  • However, the ContextStorage.addWrapper ignores any wrappers if the OpenTelemetry context storage classes have already been initialized.
  • Because Google Cloud Logging client now initializes the OpenTelemetry context storage classes early, the EventPublishingContextWrapper instance is ignored and it never wraps the OpenTelemetry ContextStorage. So EventPublishingContextWrapper is never used, and thus it will never generate any events.
  • Without the events from the EventPublishingContextWrapper, the event listening methods Slf4JEventListener and Slf4JBaggageEventListener are never triggered. Thus, tracing data is never added and removed from the logging MDC.

I created a reproduction of this problem with the following test class: https://github.com/mzeijen/spring-boot-otel-context-init-issue/blob/main/src/test/java/com/example/demo/TraceEventsTest.java

Note that this reproduction does not actually use the Google Cloud Logging client library, but it simulates is similar behaviour where a OpenTelemetry Span.current() is executed before Spring Boot can initialize OpenTelemetry.

So that we do not hit this problem we currently downgraded to Google Cloud Logging client 3.18.0, but I am also working on another workaround that will allow us to upgrade the library again. If I get it to work (it is looking good), and there is interest then I would be happy t share it.

Comment From: mhalbritter

Likely related to https://github.com/spring-projects/spring-boot/issues/41439

Comment From: mhalbritter

Another workaround would be to manually register the Slf4JEventListener and the Slf4JBaggageEventListener on the ContextStorage.

Btw, this should be a problem for users not using Spring Boot, right? If Cloud logging is triggered that early, everything relying on ContextStorageWrappers is going to fail.

I'm not sure that there's much we can do here. The logging system in Boot is initialized really early, and that apparently triggers a call to Otel.

@mzeijen What do you think about raising an issue against Google Cloud Logging?

Comment From: mzeijen

@mhalbritter How would you register the the Slf4JEventListener and the Slf4JBaggageEventListener on the ContextStorage?

Regarding raising an issue against Google Cloud Logging: I don't believe Google Cloud Logging is doing anything wrong here. They are using the OpenTelemetry classes as they are intended to be used. IMHO it is the responsibility of Spring Boot to initialize Otel correctly and sadly there is this part of Otel that needs to be initialized very early and currently it isn't initialized early enough by Spring Boot.

The javadoc of ContextStorage.addWrapper also indicates it should be called as early as possible in the lifecycle of an application:

Adds the wrapper, which will be executed with the ContextStorage is first used, i. e., by calling Context. makeCurrent(). This must be called as early in your application as possible to have effect, often as part of a static initialization block in your main class

So the ContextStorage.addWrapper should be executed before even Spring Boot initializes the logging system. But that is problematic, because at that point Spring Boot does not have the pieces to provide it with a EventPublishingContextWrapper.

However, that does not mean there isn't something we can do about. If the EventPublishingContextWrapper would allow for setting the OtelTracer.EventPublisher at a later point then at construction, then you can inject EventPublishingContextWrapper as early as possible in the Spring Boot app lifecycle. Then when the OtelTracer.EventPublisher bean is created, it can be injected into the EventPublishingContextWrapper and the wrapper will be able to generate events. As long as no OtelTracer.EventPublisher is injected yet, the EventPublishingContextWrapper will only forward the attach calls and do nothing else. This also means that no events are generated yet for any attach calls, thus the MDC won't be filled, but I think that is an acceptable situation.

I implemented a similar workaround as I propose above, but at a higher level where I created a AttachableContextStorageWrapper. This wrapper is added using ContextStorage.addWrapper as early as possible, namely via a Spring Boot start event listener. This wrapper then allows another wrapper to be attached at a later point, meaning I attach the EventPublishingContextWrapper to it when the OpenTelemetryAutoconfiguration has executed. This is all more elaborate then I believe Spring Boot will have to do, but it does show it is possible to deal with this situation without requiring any changes by Google Cloud Logging (or any other library that wants to interact with Otel early).

Comment From: mhalbritter

How would you register the the Slf4JEventListener and the Slf4JBaggageEventListener on the ContextStorage?

    public static void main(String[] args) {
        OtelTracer.EventPublisher publisher = new OtelTracer.EventPublisher() {
            private final Slf4JEventListener slf4JEventListener = new Slf4JEventListener();
            private final Slf4JBaggageEventListener slf4JBaggageEventListener = new Slf4JBaggageEventListener(List.of());

            @Override
            public void publishEvent(Object event) {
                this.slf4JEventListener.onEvent(event);
                this.slf4JBaggageEventListener.onEvent(event);
            }
        };
        ContextStorage.addWrapper(new EventPublishingContextWrapper(publisher));

        SpringApplication.run(Sb41688Application.class, args);
    }

Register the stuff yourself before bootstrapping the Spring Boot application with the run method.

Comment From: philwebb

I think #41439 should fix this for Spring Boot 3.4. Unfortunately it's a little risky to backport to earlier versions so we're going to recommend using the workaround above for those.