I no longer have kafka consumer metrics in micrometer or prometheus after my SpringBoot upgrade.
It sounds similar to https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/914 but I tried what was mentioned in that ticket and did not have any luck solving my issue.
Here are my changes:
- Upgraded SpringBoot from 2.1.18 to 2.5.7.
- Upgraded Spring from 5.1.19 to 5.3.13.
- Upgraded Kafka from 2.8.0 to 2.8.1.
- Upgraded Micrometer from 1.6.9 to 1.7.6.
- Upgraded Springfox Swagger from 2.9.2 to 3.0.0.
- Added org.springframework.kafka:spring-kafka:2.7.9
- Added org.jolokia:jolokia-core:jar:1.6.2
- Removed spring-cloud since we're not loading config from a server
- Added spring.jmx.enabled:true to my application.yml
Comment From: wilkinsona
We're not aware of any problems with Kafka metrics in Spring Boot 2.5.7. If you would like us to spend some time investigating, please spend some time providing a complete yet minimal sample that reproduces the problem. You can share it with us by pushing it to a separate repository on GitHub or by zipping it up and attaching it to this issue.
Comment From: dlipofsky
I have created a very simple example. I can confirm this works - SpringBoot 2.2.13.RELEASE - kafka-clients 2.8.1 - micrometer 1.7.6 - spring.jmx.enabled=true - spring-kafka and jolokia-core are not needed, it works without them. I tried adding them in a few combinations below because of hints on the other ticket that they may help, but they didn't.
But I tried all of these combinations and they do not work - SpringBoot 2.3.0.RELEASE, micrometer 1.7.6, no jolokia, no spring-kafka - SpringBoot 2.3.12.RELEASE, micrometer 1.7.6, no jolokia, no spring-kafka - SpringBoot 2.4.13, micrometer 1.7.6, no jolokia, no spring-kafka - SpringBoot 2.5.7, micrometer 1.7.6, no jolokia, no spring-kafka - SpringBoot 2.5.7, micrometer 1.7.6, jolokia-core 1.6.2, spring-kafka 2.7.9 - SpringBoot 2.6.1, micrometer 1.8.1 (does not work with micrometer 1.7.6), no jolokia, no spring-kafka - SpringBoot 2.6.1, micrometer 1.8.1, jolokia-core 1.7.1, spring-kafka 2.8.0
I tested all these by running the producer and confirming in the consumer logs that it had consumed messages (because it may be necessary to actually consume to create these metrics), then running
- curl http://localhost:8080/manage/prometheus
- curl http://localhost:8080/manage/metrics | jq
My sample code: - example-consumer-2.5.7.zip - example-consumer-2.2.13.zip - example-producer.zip
Comment From: dlipofsky
To clarify, I am manually creating my consumers and streams (not using spring-kafka)
and I can get metrics back by manually calling
new KafkaStreamsMetrics(stream).bindTo(registry) or
new KafkaClientMetrics(consumer).bindTo(registry)
but it would still be nice to have this happen automatically like it did in SpringBoot 2.2.x.
I've got a lot of apps to updates, and modifying each one at a time is a lot of work, and is how mistakes are made.
This is a regression, because it worked this way in SpringBoot 2.2.x but not in 2.3.x or beyond.
Comment From: ivan-zaitsev
Since Spring Boot 2.3 Kafka metrics names have been changed.
kafka_consumer_records_consumed_total_records_total -> kafka_consumer_fetch_manager_records_consumed_total
kafka_consumer_records_lag_records -> kafka_consumer_fetch_manager_records_lag
kafka_consumer_records_lag_max_records -> kafka_consumer_fetch_manager_records_lag_max
kafka_consumer_fetch_latency_max_seconds -> kafka_consumer_fetch_manager_fetch_latency_max
kafka_consumer_bytes_consumed_total_bytes_total -> kafka_consumer_fetch_manager_bytes_consumed_total
kafka_consumer_records_per_request_avg_records -> kafka_consumer_fetch_manager_records_per_request_avg
kafka_consumer_heartbeat_rate_heartbeats -> kafka_consumer_coordinator_heartbeat_rate
If you have custom consumer factories, add customizers.orderedStream().forEach(customizer -> customizer.customize(consumerFactory));
@Bean
public ConsumerFactory<Integer, String> customConsumerFactory(ObjectProvider<DefaultKafkaConsumerFactoryCustomizer> customizers) {
DefaultKafkaConsumerFactory<Integer, String> consumerFactory = new DefaultKafkaConsumerFactory<>(customConsumerConfigs());
customizers.orderedStream().forEach(customizer -> customizer.customize(consumerFactory));
return consumerFactory;
}
Comment From: dlipofsky
@ivan909020 Those factories are spring-kafka and I'm not using spring-kafka, just new KafkaConsumer(...) or new KafkaStreams(...).
I noticed the name change, which is surprising. Why did the names change with SpringBoot? I would think it would be based on the version of kafka itself or the version of micrometer, but it's not. It's frustrating since I have a bunch of established dashboards and alerts in datadog.
Comment From: ivan-zaitsev
@dlipofsky Ok, then you can create KafkaConsumer through Spring - DefaultKafkaConsumerFactory.
@Autowired
private final PrometheusMeterRegistry meterRegistry;
DefaultKafkaConsumerFactory<String, String> consumerFactory = new DefaultKafkaConsumerFactory<>(props);
consumerFactory.addListener(new MicrometerConsumerListener<>(meterRegistry));
Consumer<String, String> consumer = consumerFactory.createConsumer();
Comment From: dlipofsky
@ivan909020 , the point is I am trying to not rewrite all my apps to use spring-kafka, I would like to just be able to upgrade spring-boot (for security reasons) but I have this regression causing me pain. Also I have one app where the KafkaConsumer is created by a 3rd party SDK and therefore out of my control.
Comment From: ivan-zaitsev
@dlipofsky Ok, then you can create KafkaClientMetrics and bind to PrometheusMeterRegistry.
@Autowired
private final PrometheusMeterRegistry meterRegistry;
Consumer<String, String> consumer = new KafkaConsumer<>(props);
KafkaClientMetrics metrics = new KafkaClientMetrics(consumer);
metrics.bindTo(meterRegistry);
Comment From: dlipofsky
Yes, that is what I am doing (see my comment above from yesterday) but it still involves changing all my apps. At this point I've done it, so that ship has sailed for me, but this is still a regression and I'd like to see if it can be addressed, since the old way was better. Maybe it can help others.
Comment From: ivan-zaitsev
Yes, I see. I'm sorry I didn't notice. I suppose this is because prior to Spring Boot 2.3 Kafka metrics worked through JMX which implementation inside of KafkaConsumer and for this you did not need to configure anything, but since 2.3 it works through KafkaClientMetrics, and for this you need to configure it manually.
Comment From: snicoll
@dlipofsky this change is covered in the release notes for 2.3 that's out of OSS support by then. Not relying on JMX is important as it was disabled by default in the previous version and that led to issue there as well.
Like you said the ship has sailed here but I can imagine that it would be fairly easy for you to retrofit that behavior in a custom arrangement of yours.
Comment From: dlipofsky
@snicoll , this really is not covered by the release notes. The release notes just assume that you are using spring-kafka and only discuss what to do if you are using a custom factory. I am not using a custom factory, or any factory at all, since I am not using spring-kafka.
Comment From: snicoll
Kafka metrics are published natively for the consumers and producers created by the auto-configured ConsumerFactory and ProducerFactory.
You're right that the transition from looking at JMX metrics to using a more robust native arrangement is relying on spring-kafka. That being said, I believe you might be assuming that Spring Boot should offer you features related to Kafka, even if you're not using Spring Kafka and that part isn't accurate. Read the first sentence of the related section in the reference documentation.