Automate Service Discovery: Seamlessly Connect Prometheus with Consul

In modern micro‑service environments, the number of services grows rapidly, making static Prometheus target configuration impractical. This guide shows how to integrate Prometheus with Consul for automatic service discovery, covering ACL policy creation, token generation, adding static scrape configs via the Prometheus Operator, and verifying the setup.

1. Create minimal‑permission ACL policy

$ consul acl policy create \
  -token dfa12077-35ad-b1b2-98f9-9213ff52e20d \
  -name monitor \
  -description "It is used to monitor ACL policies and grant the service permission that starts with mon_" \
  -rules 'service_prefix "mon_" { policy = "read" } node_prefix "" { policy = "read" } agent { policy = "read" }'
ID:           fb9c1c5f-d0ec-6202-fdf1-e96d84e7759c
Name:         monitor
Description:  It is used to monitor ACL policies and grant the service permission that starts with mon_
Datacenters: 
Rules:
service_prefix "mon_" { policy = "read" }
node_prefix "" { policy = "read" }
agent { policy = "read" }

2. Create token

$ consul acl token create \
  -token dfa12077-35ad-b1b2-98f9-9213ff52e20d \
  -description "Prometheus Token" \
  -policy-name monitor
AccessorID:       6af92acf-84e9-4618-483c-2ab32f275dec
SecretID:         d56fe9fa-419f-3e44-d875-8a84508e475f
Description:      Prometheus Token
Local:            false
Create Time:      2025-06-30 07:16:23.753755553 +0000 UTC
Policies:
   fb9c1c5f-d0ec-6202-fdf1-e96d84e7759c - monitor

3. Add Prometheus static configuration

$ cat prometheus-additional.yml
- job_name: consul-discovery
  consul_sd_configs:
  - server: consul-server.consul.svc:8500
    token: d56fe9fa-419f-3e44-d875-8a84508e475f
    services: []

$ kubectl -n obs-system create secret generic additional-scrape-configs --from-file prometheus-additional.yml
secret/additional-scrape-configs created

4. Configure Prometheus Operator to use the additional scrape config

$ k -n obs-system edit prometheus monitor
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: monitor
  namespace: obs-system
spec:
  additionalScrapeConfigs:
    key: redis.yaml
    name: additional-scrape-configs
    ...

5. Verify collection

Check the Prometheus UI for the new target and confirm that the expected metrics appear.

Conclusion

Combining Prometheus with Consul enables automated, real‑time monitoring of services in cloud‑native deployments, reducing manual configuration and improving reliability.