Hi,
Summary: Using Spring boot 2.7 and Spring Cloud I can define remote baggage and when I pass in a request header of the same name this gets propagated through the app. Spring Boot 3 no longer maps the request header.
Detail:
I have a prototype to illustrate this https://github.com/davidmelia/spring-boot3-remote-baggage
In spring boot 2.7 + spring cloud sleuth if I define
spring.sleuth.baggage.correlation-fields=Customer-Name
spring.sleuth.baggage.remote-fields=Customer-Name
and pass in the request header "Customer-Name" then I can easily retrieve this header using
return Mono.just(tracer.getAllBaggage()).map(allBaggage -> {
log.info("allBaggage {}", allBaggage);
return allBaggage.get("Customer-Name") == null ? "" : allBaggage.get("Customer-Name");
}).map(customerName -> ok(Map.of("customerName", customerName))).doOnSuccess(r -> log.info("exit addressSearch()", r));
(see passing test https://github.com/davidmelia/spring-boot3-remote-baggage/blob/master/remotebaggage-boot2.7/src/test/java/com/example/demo/MicroserviceBApplicationTests.java)
In spring boot 3.0 defining
management.tracing.baggage.correlation.fields=Customer-Name
management.tracing.baggage.remote-fields=Customer-Name
and pass in the request header "Customer-Name" then this value cannot be retrieve using
return Mono.deferContextual(contextView -> {
try (ContextSnapshot.Scope scope = ContextSnapshot.setThreadLocalsFrom(contextView, ObservationThreadLocalAccessor.KEY)) {
log.info("enter addressRetrieve():: {}", addressId);
return Mono.just(tracer.getAllBaggage()).map(allBaggage -> {
log.info("allBaggage {}", allBaggage);
return allBaggage.get("Customer-Name") == null ? "" : allBaggage.get("Customer-Name");
}).map(customerName -> ok(Map.of("customerName", customerName))).doOnSuccess(r -> log.info("exit addressSearch()", r));
}
});
nor is it logged out when i define
logging.pattern.level=%5p [aid=${spring.application.name:-},tid=%X{traceId:-},sid=%X{spanId:-},cusname=%X{Customer-Name:-}]
(see failing test https://github.com/davidmelia/spring-boot3-remote-baggage/blob/master/remotebaggage-boot3.0/src/test/java/com/example/demo/MicroserviceBApplicationTests.java)
Comment From: mhalbritter
Have you seen those two samples from Micrometer, which use remote fields?
- https://github.com/micrometer-metrics/micrometer-samples/tree/main/baggage-consumer
- https://github.com/micrometer-metrics/micrometer-samples/tree/main/baggage-producer
Myabe that can help you to debug the issue.
Comment From: davidmelia
@mhalbritter I have seen those but they are not WebFlux however the samples do not work either. I have distilled the project down to prove this doesn't work even in the samples:
package com.example.micrometer;
import io.micrometer.tracing.Tracer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
@SpringBootApplication
public class BaggageConsumerApplication {
public static void main(String... args) {
new SpringApplication(BaggageConsumerApplication.class).run(args);
}
}
@RestController
class BaggageController {
private static final Logger log = LoggerFactory.getLogger(BaggageController.class);
private final Tracer tracer;
BaggageController(Tracer tracer) {
this.tracer = tracer;
}
@GetMapping("/")
public String span() {
log.info("Contains the following baggage {}", this.tracer.getAllBaggage());
String traceId = this.tracer.currentSpan().context().traceId();
log.info("<ACCEPTANCE_TEST> <TRACE:{}> Hello from producer", traceId);
return traceId;
}
}
and running
curl http://localhost:7200 -H "myremotefield: my-remote-field-value" -H "mybaggage: my-baggage"
you can see in the logs that there is no baggage
c.example.micrometer.BaggageController : Contains the following baggage {}
do you have any pointers which classes are actually responsible for converting HTTP headers into baggage?
Comment From: mhalbritter
Baggage propagation works for me. With this curl request I get the baggage in the log:
Note: this uses W3C propagation.
curl -H "mybaggage: foo" -H "traceparent: 00-6399d8fac7c5a07be596ccc41f5d058c-3e95ceefe0d3af70-01" http://localhost:8080 -s -S -v -H "Accept: application/json, */*"
will produce this log:
2022-12-14T15:09:18.994+01:00 INFO [,6399d8fac7c5a07be596ccc41f5d058c,2f6e5db07509e2bf,foo] 8051 --- [nio-8080-exec-4] com.example.gh33516.BaggageController : Contains the following baggage UnsafeArrayMap{mybaggage=foo}
2022-12-14T15:09:18.994+01:00 INFO [,6399d8fac7c5a07be596ccc41f5d058c,2f6e5db07509e2bf,foo] 8051 --- [nio-8080-exec-4] com.example.gh33516.BaggageController : <TRACE:6399d8fac7c5a07be596ccc41f5d058c> Hello from producer
with this application.yaml:
management:
tracing:
sampling:
probability: 1.0
baggage:
remote-fields:
- mybaggage
- myremotefield
correlation:
fields:
- mybaggage
propagation:
type: w3c
logging.pattern.level: "%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-},%X{mybaggage:-}]"
Note: if you don't include the traceparent header, the baggage is ignored, too.
The class which configures the propagation is org.springframework.boot.actuate.autoconfigure.tracing.BraveAutoConfiguration.BraveBaggageConfiguration
If you use B3 propagation (like Sleuth did by default, IIRC), it works with this curl:
curl -H "b3: 6399da1b314c0e4a41a4c3d715462905-1e53665b37b5fa65-1" -H "mybaggage: foo" http://localhost:8080 -s -S -v -H "Accept: application/json, */*"
Note: if you don't include the b3 header, the baggage is ignored, too.
If this doesn't work in your case, can you please share the minimal application that doesn't work?
Comment From: davidmelia
@mhalbritter thanks for that analysis - therefore I think my issue / bug / feature request is summarised by:
Using Sleuth you didn't need to pass in the traceparent with baggage so when migrating to Spring Boot 3 this is a breaking change. We have many third parties which pass in baggage who will not be able to pass in a traceparent so this is a big problem for me. Could I ask for the ability to either make traceparent optional by default to mimic sleuth or provide a feature to do this thus preserving the way Spring Cloud Sleuth handles baggage?
I have a project to illustrate this
If you run https://github.com/davidmelia/spring-boot3-remote-baggage/tree/master/remotebaggage-boot2.7 and curl
curl http://localhost:9081/addresses/1234 -H "customer-name: Dave"
then you get back the name
{"customerName":"Dave"}
If you run my Spring Boot 3 example https://github.com/davidmelia/spring-boot3-remote-baggage/tree/master/remotebaggage-boot3.0 curling does not work
curl http://localhost:9081/addresses/1234 -H "customer-name: Dave"
which doesn't work
{"customerName":""}
and as you say you have to pass in the tradeparent which is a breaking change compared with sleuth
curl http://localhost:9081/addresses/1234 -H "Customer-Name: Dave" -H "traceparent: 00-6399d8fac7c5a07be596ccc41f5d058c-3e95ceefe0d3af70-01"
{"customerName":"Dave"}
Thanks
Comment From: mhalbritter
Chiming in @marcingrzejszczak - what do you think?
Comment From: marcingrzejszczak
So that seems like a bug in micrometer tracing. Currently, for OTel bridge it works only with tracing data being there in the headers. Can you try doing the same exercise but change the propagation type to B3 instead of the default W3C ?
Comment From: mhalbritter
B3 also needs the b3 header, otherwise the baggage is not visible to the application.
Comment From: davidmelia
@marcingrzejszczak @mhalbritter I can confirm b3 has the same behaviour
curl http://localhost:9081/addresses/1234 -H "Customer-Name: Dave" -H "b3: 6399da1b314c0e4a41a4c3d715462905-1e53665b37b5fa65-1"
works fine
{"customerName":"Dave"}
but without the b3 header it doesn't
curl http://localhost:9081/addresses/1234 -H "Customer-Name: Dave"
{"customerName":""}
Comment From: marcingrzejszczak
Let's close this one in favour of https://github.com/micrometer-metrics/micrometer/issues/3564