There are multiple solutions on reporting Kafka offset information, such as Kafka distribution’s ConsumerOffsetChecker or Linkedin’s Burrow (which is more advanced and has an alerting and recovery built-in logic). In my company we were already using Quantifind’s KafkaOffsetMonitor. It’s a very useful visualization tool for Kafka topic and their consumers. It visualizes the relation between topics and consumer groups and allows to drill down on a consumer group level and review the offset over time. It lets the user decide whether there is an issue with the lags. It’s definitely not only useful for developers!
In the current code version it is possible to report those visualized offsets to any reporting system, by plugging in reporters. This reporters are called periodically by the application. In order for these reports to be discovered, the Jar containing the implemented reporter needs to be added to the classpath.
A graphite reporting tool is already existing, but since we are using Datadog we needed to do have an appropriate plugin. Leveraging again Coursera’s Datadog reporter, as I have done in a previous post, we are able to use MetricRegistry and report via StatsD.
In my implementation, I am registering gauges once and keeping the concrete values in a map. I did not spot any concurrency dangers as it seems that the offset information cannot be modified concurrently, since the reporters are called sequentially by the application. Hence, the gauge values are saved in a regular Map and not in a concurrent map.
Unfortunately an official release of Quantafind’s KafkaOffsetMonitor supporting pluggable offset reporters is not existing at the moment, therefore one must build it from source code, or use the one I build (on Ubuntu 16.04 LTS) and packed into the project. Also it seems like the project development is stale. We are using an older Kafka version (0.8.2) and that could be a reason why we have not encountered issues with it.
Getting it:
The code (with the built Jar) can be viewed here.