Extracted from https://github.com/spring-projects/spring-boot/issues/29666

Micrometer Tracing comes with the following features

  • Abstraction over a Tracer (Tracer is a library handling a lifecycle of a span. Span is wrapping an action that we would like to measure. Span contains information such as timing, span id - unique identifier for an action, trace id - same identifier for all actions within the same business transaction)
  • Tracer bridges. We're bridging from the abstraction to a concrete tracer implementation. We support the following tracers
  • Brave
  • OpenTelemetry (OTel)
  • TracingObservationHandler interface and its implementations
  • Span exporters /reporters (to visualize latency we should send spans to a reporting system such as Wavefront)
  • Zipkin with Brave
  • Anything that OTel provides out of the box (Zipkin, OTLP etc.)
  • Wavefront with Brave and OTel

We need Boot to configure the Bridges with all of their features.

Comment From: mhalbritter

Base infrastructure

  • A lot of the auto-configurations can be used from Spring Cloud Sleuth: brave and otel
  • We need to create a Micrometer DefaultTracingObservationHandler bean
  • This will get picked up by the auto-configuration from https://github.com/spring-projects/spring-boot/issues/29666
  • This needs a Micrometer Tracer. There are two tracer implementations shipped with Micrometer, BraveTracer and OtelTracer

Brave

  • Code in micrometer
  • To create a Micrometer BraveTracer, we need a brave Tracer and some other stuff, which all needs Brave Tracing
  • Tracing can be customized with some options like the local service name, etc. The reported spans are handled by brave SpanHandlers.

Brave & Zipkin

Zipkin is implemented as a Brave SpanHandler

var sender = URLConnectionSender.create("http://127.0.0.1:9411/api/v2/spans");
var spanReporter = AsyncReporter.builder(sender).build();
var spanHandler = new ZipkinSpanHandler.newBuilder(spanReporter).alwaysReportSpans(true).build();

I guess there are a lot more of the sender implementations.

Brave & Wavefront

  • Wavefront is implemented as a brave SpanHandler, see here

Open Telemetry

Open Telemetry & Wavefront

  • Wavefront is implemented a a otel SpanExporter, see here

Comment From: mhalbritter

What's done as of now:

Micrometer tracing

  • If micrometer-tracing is on the classpath, the DefaultTracingObservationHandler is registered with the ObservationRegistry. This creates spans when an Observation ends.

Brave

  • If brave is on the classpath: Tracing, Tracer, CurrentTraceContext, B3 propagation, configurable sampling (percent based)
  • If micrometer-tracing-bridge-brave is on the classpath: BraveTracer and BraveBaggageManager (BraveTracer is then consumed by DefaultTracingObservationHandler). This bridges spans created via ending an Observation through the handler to Brave.

OpenTelemetry

  • if opentelemetry-api is on the classpath: Tracer (needs OpenTelemetry bean, which pops up as soon as opentelemetry-sdk-trace is on the classpath, see below)
  • if opentelemetry-sdk-trace is on the classpath: OpenTelemetry, SdkTracerProvider (is consumed by OpenTelemetry), ContextPropagators, Sampler with configurable sampling (percent based), SpanProcessor
  • if micrometer-tracing-bridge-otel is on the classpath: EventPublisher (noop implementation), OtelCurrentTraceContext, OtelTracer (which is consumed by DefaultTracingObservationHandler). This bridges spans created via ending an Observation through the handler to OpenTelemetry.

Zipkin

  • If zipkin-reporter is on the classpath: An URLConnectionSender with a configurable endpoint, a JSON_V2 span encoder, an RestTemplateSender with a configurable endpoint, AsyncReporter

Zipkin via Brave

  • If zipkin-reporter-brave is on the classpath: ZipkinSpanHandler, which is consumed by braves Tracing (that means that all spans reported to brave will be sent to zipkin)

Zipkin via Open Telemetry

  • If opentelemetry-exporter-zipkin is on the classpath: ZipkinSpanExporter, which is consumed by otel SpanProcessor (that means that all spans reported to OpenTelemetry will be sent to Zipkin)

Wavefront

  • if wavefront-sdk-java is on the classpath: WavefrontSender (with configurable URL, source, api token, service name and sender properties like queue size, etc.), ApplicationTags
  • if micrometer-tracing-reporter-wavefront is on the classpath: WavefrontSpanHandler

Wavefront Micrometer Metrics

  • The wavefront sender reports metrics for sent, dropped, error spans. If micrometer-core is on the classpath: MeterRegistrySpanMetrics which reports metrics via MeterRegistry. if it's not on the classpath, NoopSpanMetricsConfiguration steps in.

Wavefront via Brave

  • if micrometer-tracing-reporter-wavefront and brave is on the classpath: WavefrontBraveSpanHandler which exports to wavefront via brave

Wavefront via OpenTracing

  • if micrometer-tracing-reporter-wavefront and opentelemetry is on the classpath: WavefrontOtelSpanHandler which exports to wavefront via opentelemetry

Comment From: mhalbritter

Open TODOs & questions

TODOs

  • Spring Boot already defines a WavefrontSender bean, configurable via WavefrontProperties. they are configured via management.metrics.export.wavefront prefix. We should merge this with tracing.
  • openTelemetry.getTracer(...) wants a name, what should we provide here? org.springframework.boot?

Questions

  • What feature set do we want to support? There is a near infinite number of reporting backends and transports to use

Comment From: FranPregernik

Hi!

We have begun integrating Actuator OpenTelemetry Metrics and Tracing into a project of ours. I have a few observations/suggestions I would like to propose.

As a starting point, the tracing OpenTelemetryAutoConfiguration sets up an instance of the OpenTelemetry. It is a great way to specify a non-supported exporter (e.g. OtlpGrpcSpanExporter).

But the OtlpMetricsExportAutoConfiguration does not use the OpenTelemetry instance at all and registers the OtlpMeterRegistry based on the configuration. Currently I have no way of specifying a OtlpGrpcMetricExporter that it will use.

My proposition is to have separate OpenTelemetry autoconfigure, a OpenTelemetry tracing autoconfigure and OpenTelemetry metrics autoconfigure.

The OpenTeleletry configuration would have the following:

    @Bean
    @ConditionalOnMissingBean
    fun otelResource(environment: Environment): Resource {
        val applicationName = environment.getProperty("spring.application.name", "application")
        return Resource.create(Attributes.of(ResourceAttributes.SERVICE_NAME, applicationName))
    }

    @Bean
    @ConditionalOnMissingBean
    fun openTelemetry(
        sdkTracerProvider: ObjectProvider<SdkTracerProvider>,
        sdkMeterProvider: ObjectProvider<SdkMeterProvider>,
        contextPropagators: ObjectProvider<ContextPropagators>
    ): OpenTelemetry {
        val builder = OpenTelemetrySdk.builder()
        contextPropagators.ifUnique {
            builder.setPropagators(it)
        }
        sdkMeterProvider.ifUnique {
            builder.setMeterProvider(it)
        }
        sdkTracerProvider.ifUnique {
            // set by
            // org.springframework.boot.actuate.autoconfigure.tracing.OpenTelemetryAutoConfiguration.otelSdkTracerProvider
            builder.setTracerProvider(it)
        }
        return builder.build()
    }

The otelResource is a consistent way to specify the service name and other parameters for both metrics and tracing. The OpenTelemetry accepts the additional sdkMeterProvider and allows a setup of a custom MetricExporter.

The OtlpMetricsExportAutoConfiguration would create the SdkMeterProvider:

    @Bean
    fun sdkMeterProvider(
        metricExportersProvider: ObjectProvider<List<MetricExporter>>,
        otelResource: Resource
    ): SdkMeterProvider {
        // from https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/spring/spring-boot-autoconfigure/src/main/java/io/opentelemetry/instrumentation/spring/autoconfigure/OpenTelemetryAutoConfiguration.java
        val meterProviderBuilder = SdkMeterProvider.builder()
        val interval = properties.metrics.interval
        metricExportersProvider.getIfAvailable { emptyList() }
            .map { metricExporter ->
                val metricReaderBuilder = PeriodicMetricReader.builder(metricExporter)
                if (interval != null) {
                    metricReaderBuilder.setInterval(interval)
                }
                metricReaderBuilder.build()
            }
            .forEach { reader ->
                meterProviderBuilder.registerMetricReader(reader)
            }
        return meterProviderBuilder.setResource(otelResource).build()
    }

And setup the registry in OtlpMetricsExportAutoConfiguration like so:

    @Bean
    fun otelMeterRegistry(openTelemetry: OpenTelemetry): MeterRegistry {
        return OpenTelemetryMeterRegistry.create(openTelemetry)
    }

    @Bean
    @ConditionalOnMissingBean(MetricExporter::class)
    fun metricExporter(...) : OtlpHttpMetricExporter {
       // ...
    }

So now both tracing and metrics use the same OpenTelemetry instance that is set up with a single OpenTelemetry Resource instance. The default metric exporter is the OtlpHttpMetricExporter and can be overridden with the GRPC version.

Comment From: mhalbritter

Hi @FranPregernik, as this is a closed issue, could you please create your enhancement ideas in a separate, new issue? Thanks!