Blackbox Monitoring with Prometheus Blackbox Exporter in Kubernetes

In addition to the white‑box monitoring covered previously (resource usage, container status, service discovery, etc.), a complete observability strategy should also include black‑box monitoring, which tests services from an external user perspective using probes such as HTTP, TCP, DNS, and ICMP.

The Prometheus community provides the official blackbox_exporter to perform these external checks. It supports probing via HTTP, HTTPS, DNS, TCP, and ICMP, and can be configured per module.

First, deploy the exporter in the cluster and provide its configuration through a ConfigMap . An example configuration (saved as prome-blackbox.yaml ) is:

apiVersion: v1
kind: ConfigMap
metadata:
  name: blackbox-config
  namespace: kube-mon

data:
  blackbox.yml: |
    modules:
      http_2xx:
        prober: http
        timeout: 10s
        http:
          valid_http_versions: ["HTTP/1.1", "HTTP/2"]
          valid_status_codes: [200]
          method: GET
          preferred_ip_protocol: "ip4"
      http_post_2xx:
        prober: http
        timeout: 10s
        http:
          valid_http_versions: ["HTTP/1.1", "HTTP/2"]
          method: POST
          preferred_ip_protocol: "ip4"
      tcp_connect:
        prober: tcp
        timeout: 10s
      dns:
        prober: dns
        timeout: 10s
        dns:
          transport_protocol: "tcp"
          preferred_ip_protocol: "ip4"
          query_name: "kubernetes.default.svc.cluster.local"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blackbox
  namespace: kube-mon
spec:
  selector:
    matchLabels:
      app: blackbox
  template:
    metadata:
      labels:
        app: blackbox
    spec:
      containers:
      - image: prom/blackbox-exporter:v0.16.0
        name: blackbox
        args:
        - --config.file=/etc/blackbox_exporter/blackbox.yml
        ports:
        - containerPort: 9115
        volumeMounts:
        - name: config
          mountPath: /etc/blackbox_exporter
      volumes:
      - name: config
        configMap:
          name: blackbox-config
---
apiVersion: v1
kind: Service
metadata:
  name: blackbox
  namespace: kube-mon
spec:
  selector:
    app: blackbox
  ports:
  - port: 9115
    targetPort: 9115

Apply the resources with kubectl apply -f prome-blackbox.yaml . Then add a scrape job for the exporter in the Prometheus configuration (also stored in a ConfigMap ):

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: kube-mon

data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      scrape_timeout: 15s
    scrape_configs:
    - job_name: 'kubernetes-service-dns'
      metrics_path: /probe
      params:
        module: [dns]
      kubernetes_sd_configs:
      - role: service
      relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115
    - job_name: 'kubernetes-http-services'
      metrics_path: /probe
      params:
        module: [http_2xx]
      kubernetes_sd_configs:
      - role: service
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_http_probe]
        action: keep
        regex: true
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox:9115
    - job_name: 'kubernetes-ingresses'
      metrics_path: /probe
      params:
        module: [http_2xx]
      kubernetes_sd_configs:
      - role: ingress
      relabel_configs:
      - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_http_probe]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
        regex: (.+);(.+);(.+)
        replacement: $1://$2$3
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox:9115

After reloading Prometheus (e.g., curl -X POST http:// :9090/-/reload ), the new jobs appear on the Targets page. To enable probing for a specific Service or Ingress, add the annotation prometheus.io/http_probe="true" (and optionally prometheus.io/http_probe_port and prometheus.io/http_probe_path ) to the object's metadata.

Example Service annotation:

metadata:
  annotations:
    prometheus.io/http_probe: "true"
    prometheus.io/http_probe_port: "8080"
    prometheus.io/http_probe_path: "/healthz"

Example Ingress annotation:

metadata:
  annotations:
    prometheus.io/http_probe: "true"

With these annotations, Prometheus will automatically generate black‑box scrape jobs for the annotated objects, and metrics such as probe_success and probe_duration_seconds can be visualized in Grafana.

The Blackbox Exporter also supports TCP, DNS, and ICMP probes; refer to its GitHub repository for additional module definitions. Official example configurations for Prometheus and Blackbox Exporter can be found at the links provided in the original article.