Introduction

In an earlier article about custom metrics we explored the process of creating custom metrics in a Spring Boot Application and by the end of it we had two code examples of working metrics that could be scrapped by Prometheus via the Spring Boot actuator endpoint.

Next Steps

This time we'll make sure:

• Prometheus alerts you when dependencies or scheduled jobs fail

• Grafana shows a green/red indicator with historical trends

Prometheus Alert Rules

Save this as alerts_springboot.yml and include it in your Prometheus prometheus.yml config.

groups:
  - name: springboot-health
    interval: 30s
    rules:
      - alert: BookingsServiceDependenciesUnhealthy
        expr: bookingsService_checkDependencies == 0
        for: 2m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "Dependencies unhealthy in the Bookings service"
          description: "The bookings service dependency check has failed for more than 2 minutes."

      - alert: BookingsServiceScheduledJobsUnhealthy
        expr: bookingsService_checkScheduledJobs == 0
        for: 2m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "Scheduled jobs unhealthy in Bookings service"
          description: "Scheduled jobs are failing or delayed for more than 2 minutes."

Why this works well:

• == 0 means "not OK" in your gauge system

• for: 2m prevents flapping alerts for temporary issues

• severity levels allow routing to different channels (PagerDuty, Slack, email, etc.)

Grafana Panel --- Status + Trend

Here's a panel JSON you can import directly into Grafana.

• Step 1: Go to Grafana → Dashboard → Import → Paste JSON

• Step 2: Select your Prometheus datasource

• Step 3: Adjust titles if needed

{
  "type": "stat",
  "title": "Bookings Service Health",
  "targets": [
    {
      "expr": "bookingsService_checkDependencies",
      "legendFormat": "Dependencies",
      "refId": "A"
    },
    {
      "expr": "bookingsService_checkScheduledJobs",
      "legendFormat": "Scheduled Jobs",
      "refId": "B"
    }
  ],
  "options": {
    "reduceOptions": {
      "values": false,
      "calcs": ["lastNotNull"],
      "fields": ""
    },
    "orientation": "auto",
    "colorMode": "value",
    "graphMode": "area",
    "justifyMode": "auto",
    "displayMode": "color-background"
  },
  "fieldConfig": {
    "defaults": {
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "red", "value": 0 },
          { "color": "green", "value": 1 }
        ]
      },
      "mappings": [
        {
          "type": "value",
          "options": {
            "0": { "color": "red", "text": "Unhealthy" },
            "1": { "color": "green", "text": "Healthy" }
          }
        }
      ]
    },
    "overrides": []
  }
}

What you get in Grafana:

• Big green "Healthy" / red "Unhealthy" indicators for each metric

• A quick way to see current status at a glance

• Historical trend line (if you switch to time series mode)

Conclusion

On this article we looked into a simple example of how to combine custom metrics exposed by your application, and once they are scrapped by Prometheus we were able to make a ready for production dashboard in grafana.