Monitoring Spring Boot Application With Micrometer, Prometheus And Grafana Using Custom Metrics

In this blog post, I will demonstrate how a Spring Boot web application can be monitored using Micrometer which exposes metrics from our application, Prometheus which stores the metric data, and Grafana to visualize the data in graphs.

As always, the code for the demo used in this article can be found on GitHub.

Spring Boot

We initialized the project using spring-boot-starter-actuator which already exposes production-ready endpoints.

If we start our application we can see that some endpoints like health and info are already exposed to the /actuator endpoint per default.

Triggering the /actuator/health endpoint gives us a metric if the service is up and running:

application.yml :

server:
  ssl:
    key-store: classpath:example.jks
    key-store-password: password
  port: 8443

management:
  endpoints:
    web:
      exposure:
        include: prometheus,health,info,metric,beans

As you can i enable http on my spring boot app using Self-Signed Certificate in Spring Boot, you don’t have to do it , im doing it because im testing also some jmeter scenarios.

Micrometer

Micrometer provides a simple facade over the instrumentation clients for the most popular monitoring systems, allowing you to instrument your JVM-based application code without vendor lock-in. Think SLF4J, but for metrics.

Micrometer is an open-source project and provides a metric facade that exposes metric data in a vendor-neutral format that a monitoring system can understand. These monitoring systems are supported:

  • AppOptics
  • Azure Monitor
  • Netflix Atlas
  • CloudWatch
  • Datadog
  • Dynatrace
  • Elastic
  • Ganglia
  • Graphite
  • Humio
  • Influx/Telegraf
  • JMX
  • KairosDB
  • New Relic
  • Prometheus
  • SignalFx
  • Google Stackdriver
  • StatsD
  • Wavefront

Micrometer is not part of the Spring ecosystem and needs to be added as a dependency. In our demo application, this was already done in the Spring Initializr configuration.

Next step is to expose the Prometheus metrics in application.properties:

management.endpoints.web.exposure.include=prometheus,health,info,metric

Now we can trigger this endpoint and see the Prometheus metrics:

Custom Metrics

We can also define some custom metrics, which I will demonstrate in this section. The demo contains a Scheduler class which periodically runs the included schedulingTask method.

To be able to send custom metrics we need to import MeterRegistry from the Micrometer library and inject it into our class. For more detail please check the official documentation.

It is possible to instantiate these types of meters from MeterRegistry:

  • Counter: reports merely a count over a specified property of an application
  • Gauge: shows the current value of a meter
  • Timers: measures latencies or frequency of events
  • DistributionSummary: provides distribution of events and a simple summary

I implemented a counter and a gauge for demonstration purposes:

package com.example.demo;

import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

import java.util.Random;
import java.util.concurrent.atomic.AtomicInteger;

@Component
public class Scheduler {

    private final AtomicInteger testGauge;
    private final Counter testCounter;

    public Scheduler(MeterRegistry meterRegistry) {
        testGauge = meterRegistry.gauge("custom_gauge", new AtomicInteger(0));
        testCounter = meterRegistry.counter("custom_counter");
    }

    @Scheduled(fixedRateString = "1000", initialDelayString = "0")
    public void schedulingTask() {
        testGauge.set(Scheduler.getRandomNumberInRange(0, 100));
        testCounter.increment();
    }

    private static int getRandomNumberInRange(int min, int max) {
        if (min >= max) {
            throw new IllegalArgumentException("max must be greater than min");
        }

        Random r = new Random();
        return r.nextInt((max - min) + 1) + min;
    }
}

Prometheus

Prometheus stores our metric data in time series in memory by periodically pulling it via HTTP. The data can be visualized by a console template language, a built-in expression browser, or by integrating Grafana (which we will do after setting up Prometheus).

In this demo, we will run Prometheus locally in a Docker container and we, therefore, need some configurations in a prometheus.yml file that you can place anywhere on your hard drive:

# my global config
global:
  scrape_interval: 2s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 2s # Evaluate rules every 15 seconds. The default is 
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.

scrape_configs:
  - job_name: 'Spring Boot Application input'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 2s
    scheme: https
    static_configs:
      - targets: ['192.168.56.1:8443']
        labels:
          application: 'My Spring Boot Application'
    tls_config:
      insecure_skip_verify: true

In my case im running Prometheus using docker insalled on a VM using virtiualBox.

docker run -d -p 9090:9090 -v <path-to-your-prometheus.yml>:/etc/prometheus/prometheus.yml prom/prometheus

<path-to-your-prometheus.yml> should be the path where you placed the prometheus.yml configuration file described above.

Note that if you r doing the same you need to create some networking rules in your virtualBox so you can acces the UI web of Prometheus and Grafana in your host machine

To check that Prometheus is correctly listening to our locally running Spring Boot application we can navigate to Status -> Targets in the top main navigation bar:

Finally, we can open the Prometheus on http://192.168.56.1:9090/ in the web browser and search for our custom metric named custom_gauge:

Grafana

The included Prometheus browser graph is nice for basic visualization of our metrics but we will use Grafana instead. Grafana provides a rich UI where you create, explore and share dashboards that contain multiple graphs.

Grafana can pull data from various data sources like Prometheus, Elasticsearch, InfluxDB, etc. It also allows you to set rule-based alerts, which then can notify you over Slack, Email, Hipchat, and similar.

We start Grafana also locally in a Docker container:

docker run -d -p 3000:3000 grafana/grafana

Note that you need to add a port forwarding rule to access the UI web grafana, see above.

Opening http://192.168.56.1:3000/ in a browser should now show the following login page:

You can log in using the default username admin and the default password admin.

The first step is to add our local Prometheus as our data source:

The first dashboard we want to add is the popular JVM dashboard:

Custom Metric Dashboard

Finally, we want to create a new dashboard where we show our custom metrics. The first step is to create a new dashboard:

Now we see a new dashboard where we can create a new panel:

In the first panel we add a visualization for our custom_gauge metric. I use the Stat visualization as it shows the current value and a simple graph:

Additionally, a new panel for the custom_counter metric is added to our dashboard: