SpringBoot Inject user-defined Attributes into SdkTracerProvider

User-defined attributes is useful for tag at service level, for example Jaeger UI treat it as process tag beside span tag.

We can define a SdkTracerProviderBuilderCustomizer to customize Attributes, but it is tedious and you have to include ResourceAttributes.SERVICE_NAME again.

    @Bean
    SdkTracerProviderBuilderCustomizer sdkTracerProviderBuilderCustomizer(Environment environment) {
        return builder -> {
            String applicationName = environment.getProperty("spring.application.name", "application");
            Attributes attributes = Attributes.of(ResourceAttributes.SERVICE_NAME, applicationName,
                    AttributeKey.stringKey("java.version"), System.getProperty("java.version"));
            builder.setResource(Resource.create(attributes));
        };
    }

now you can do it like this

    @Bean
    Attributes javaAttributes() {
        return Attributes.of(AttributeKey.stringKey("java.version"), System.getProperty("java.version"));
    }

Comment From: quaff

SdkTracerProviderBuilderCustomizer

SdkTracerProviderBuilder doesn't expose method to obtain existed Resource, It means you have to inject Environment and set ResourceAttributes.SERVICE_NAME, which is done by spring boot already.

Comment From: quaff

Do you mean add default method to SdkTracerProviderBuilderCustomizer like this?

@FunctionalInterface
public interface SdkTracerProviderBuilderCustomizer {

    /**
     * Customize the given {@code builder}.
     * @param builder the builder to customize
     */
    void customize(SdkTracerProviderBuilder builder);

    /**
     * Customize the given {@code builder}.
     * @param builder the builder to customize
     */
    default void customize(AttributesBuilder builder) {

    }
}

        String applicationName = environment.getProperty("spring.application.name", DEFAULT_APPLICATION_NAME);
        AttributesBuilder attributesBuilder = Attributes.builder();
        attributesBuilder.put(ResourceAttributes.SERVICE_NAME, applicationName);
        customizers.orderedStream().forEach((customizer) -> customizer.customize(attributesBuilder));

        @Bean
        SdkTracerProviderBuilderCustomizer sdkTracerProviderBuilderCustomizer() {
            return new SdkTracerProviderBuilderCustomizer() {

                @Override
                public void customize(SdkTracerProviderBuilder builder) {

                }

                @Override
                public void customize(AttributesBuilder builder) {
                    builder.put(AttributeKey.stringKey("java.version"), System.getProperty("java.version"));
                }
            };
        }

Comment From: wilkinsona

Yes, that's what I meant. It's somewhat similar to what we've done in org.springframework.boot.autoconfigure.elasticsearch.RestClientBuilderCustomizer for example.

Comment From: quaff

Yes, that's what I meant. It's somewhat similar to what we've done in org.springframework.boot.autoconfigure.elasticsearch.RestClientBuilderCustomizer for example.

Updated as your suggestion.

Comment From: jonatan-ivanov

@quaff Did you check the ObservationFilter component? With it you can add KeyValues that will translate to Attributes. It has two advantages: - It does not depend on OTel so it makes easier if you want to migrate - You have a control to set attributes to metrics-only or both metrics and spans (or other things you can come up with)

I don't think it defeats the purpose of this customizer (e.g.: if you use OTel directly) but might help.

Comment From: wilkinsona

Thanks again for the PR, @quaff, and for the updates as well. Unfortunately, we discussed this one today and concluded that we don't have enough information to make a change in this area. It's not clear to use if the customization should be at the Attributes level or the Resource level. The latter would allow the schema URL to be specified. Merging two Resource instances where one has a URL and one does not results in the URL being dropped. This complicates provided default attributes if we allow customization at the Resource level. For the time being, we think that the duplication of the code to set the SERVICE_NAME attribute is the least-bad option. We can reconsider this in the future if a clear pattern for the required level of customization emerges.

Comment From: quaff

@quaff Did you check the ObservationFilter component? With it you can add KeyValues that will translate to Attributes. It has two advantages:

It does not depend on OTel so it makes easier if you want to migrate

You have a control to set attributes to metrics-only or both metrics and spans (or other things you can come up with)

I don't think it defeats the purpose of this customizer (e.g.: if you use OTel directly) but might help.

@jonatan-ivanov ObservationFilter works, can you give me a hint how to control to set attributes to metrics-only or both metrics and spans?

Comment From: jonatan-ivanov

By default the meter handler will use .lowCardinalityKeyValue and the tracing handler will use both low- and .highCardinalityKeyValue. So: - Both: add with .lowCardinalityKeyValue - Spans-only: add with .highCardinalityKeyValue - Metrics-only: does not exists by default, either you do both and remove tags using a SpanFilter or you override DefaultTracingHandler.

What is the use-case to make the information available only for metrics?

Comment From: quaff

By default the meter handler will use .lowCardinalityKeyValue and the tracing handler will use both low- and .highCardinalityKeyValue. So:

Both: add with .lowCardinalityKeyValue

Spans-only: add with .highCardinalityKeyValue

Metrics-only: does not exists by default, either you do both and remove tags using a SpanFilter or you override DefaultTracingHandler.

What is the use-case to make the information available only for metrics?

Thanks for your elaboration, I'd like to use MeterRegistryCustomizer to configure commonTags for metrics-only. If I understand correctly, spans-only must be added by .highCardinalityKeyValue even though it's low cardinality like java.version?

Comment From: jonatan-ivanov

Yes, Micrometer does not know the cardinality of the tags so you can add low cardinality tags as highCardinalityKeyValue (and vice-versa but if you attach high cardinality tags as lowCardinalityKeyValue, your app and your metric backend will not like the high cardinality data).

Comment From: quaff

@quaff Did you check the ObservationFilter component? With it you can add KeyValues that will translate to Attributes. It has two advantages:

It does not depend on OTel so it makes easier if you want to migrate

You have a control to set attributes to metrics-only or both metrics and spans (or other things you can come up with)

I don't think it defeats the purpose of this customizer (e.g.: if you use OTel directly) but might help.

@jonatan-ivanov tags added by ObservationFilter is span level not process level, not my desired behavior.

Comment From: jonatan-ivanov

I'm not sure I understand what you mean by span vs. process level tags.

Comment From: quaff

I'm not sure I understand what you mean by span vs. process level tags.

You can check out the screenshot image at the begin of this thread, Jaeger UI will separate process level tags from span level tags, process level tags is like commonTags of MeterRegistry

Comment From: jonatan-ivanov

Do you happen to know if Jaeger behaves the same if you use the Zipkin format? If so, I think you "just" need to take care about the naming of your tags and they should become "process" attributes. I don't know if this is an existing feature or not (should be) or what is the naming convention (if any).

Comment From: amithkumarg

Do you happen to know if Jaeger behaves the same if you use the Zipkin format? If so, I think you "just" need to take care about the naming of your tags and they should become "process" attributes. I don't know if this is an existing feature or not (should be) or what is the naming convention (if any).

@jonatan-ivanov This is where the mapping between OpenTelemetry & Jaeger is defined. https://www.jaegertracing.io/docs/1.45/architecture/#terminology You can see the Jaeger Process tags comes out of OpenTelemetry Resource and Span tags maps from OpenTelemetry attributes.

Comment From: quaff

Do you happen to know if Jaeger behaves the same if you use the Zipkin format? If so, I think you "just" need to take care about the naming of your tags and they should become "process" attributes. I don't know if this is an existing feature or not (should be) or what is the naming convention (if any).

I don't think it related to naming, jaeger have tracer level tag since OpenTracing

https://github.com/jaegertracing/jaeger-client-java/blob/master/jaeger-core/src/main/java/io/jaegertracing/internal/JaegerTracer.java#L554

Comment From: jonatan-ivanov

@amithkumarg

This is where the mapping between OpenTelemetry & Jaeger is defined.

Please check my question again, I was asking about the Zipkin format not OpenTelemetry. OTel is easier to handle for Jaeger since it has the concept of Resources, Zipkin does not.

@quaff

I don't think it related to naming, jaeger have tracer level tag since OpenTracing

That's fine, I'm asking about Zipkin since it does not have anything just tags on the spans. If Jaeger implemented handling this (I hope they did), you should be able to piggyback on this behavior.

Comment From: quaff

@amithkumarg

This is where the mapping between OpenTelemetry & Jaeger is defined.

Please check my question again, I was asking about the Zipkin format not OpenTelemetry. OTel is easier to handle for Jaeger since it has the concept of Resources, Zipkin does not.

@quaff

I don't think it related to naming, jaeger have tracer level tag since OpenTracing

That's fine, I'm asking about Zipkin since it does not have anything just tags on the spans. If Jaeger implemented handling this (I hope they did), you should be able to piggyback on this behavior.

Sorry I'm not familiar with Zipkin, but I can verify that Jaeger use Resource of SdkTracerProvider as process level tag, that's why I'm open this PR. I'm turning to customize SdkTracerProvider, just let you know that ObservationFilter doesn't satisfy.