How to Monitor Hybrid Cloud with Prometheus: Fix Missing Physical Machine Targets

Problem Scenario Reconstruction

A user wants a single Prometheus job to monitor both Kubernetes‑deployed

node-exporter

(exposed via endpoints) and a directly deployed

node-exporter

on a physical machine (IP:Port).

Initial Configuration

<code>- job_name: node-exporter
  static_configs:
  - targets: ["172.139.20.1:9100"] # physical machine address
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - action: keep
    source_labels: [__meta_kubernetes_namespace,__meta_kubernetes_endpoints_name,__meta_kubernetes_endpoint_port_name]
    regex: kube-system;node-exporter;metrics
</code>

Result: the physical‑machine target 172.139.20.1:9100 could not be scraped.

Root Cause Analysis

Prometheus merges all discovery mechanisms (static_configs + kubernetes_sd_configs) under the same job.

The

keep

action in

relabel_configs

requires targets to match the Kubernetes metadata label

kube-system;node-exporter;metrics

.

Physical‑machine targets lack any

__meta_kubernetes_*

labels, so they are filtered out.

Solution: Separate Handling + Dynamic Labeling

Core idea: use relabel rules to distinguish Kubernetes and physical‑machine targets and treat them separately.

Optimized Configuration

<code>- job_name: node-exporter
  static_configs:
  - targets: ["172.139.20.1:9100"]
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  - source_labels: [__address__]
    action: replace
    regex: (.*):10250
    target_label: __address__
    replacement: $1:9100
  - source_labels: [__address__]
    target_label: instance
</code>

Key Configuration Interpretation

Physical‑machine static config: define targets directly with

static_configs

to avoid being filtered by Kubernetes relabel rules.

Kubernetes dynamic discovery: use the

node

role to automatically discover cluster nodes.

The

replace

action changes the scraped metrics port from the default

10250

to

9100

.

Both discovery mechanisms apply a second relabel rule that copies the

__address__

value to the

instance

label.

Effect Verification

Visit the Prometheus

targets

page (e.g.,

http://<prometheus>:9090/targets

) to confirm that both the Kubernetes and physical‑machine

node-exporter

targets appear and are being scraped.

Conclusion

By applying appropriate relabel rules and label strategies, Prometheus can seamlessly monitor hybrid environments. The provided configuration has been validated in production and can be reused or adapted to fit specific labeling requirements.