Kubernetes Monitoring: Why It’s Needed, Core Components, and Metric Exposure

Kubernetes has become the de‑facto standard for container orchestration. Understanding its core principles and monitoring its components is essential for backend developers and anyone working with cloud‑native systems.

Why monitor Kubernetes? Monitoring helps you detect resource contention (CPU, memory, disk I/O), scheduling delays, network bottlenecks, and component failures (API Server, etcd, kubelet, scheduler). Without visibility, small issues can quickly turn into major outages.

Key components and metrics

API Server : request latency, request rate, error rate.

etcd : write latency, read latency, storage size, leader election frequency.

Scheduler : scheduling latency, failure rate.

Controller Manager : controller processing latency, error rate.

kubelet : pod start latency, container crash count.

kube‑proxy : service request latency, connection error rate.

Node‑level resources

CPU usage and limits.

Memory usage and OOM events.

Disk usage and I/O latency.

Network bandwidth and packet loss.

Pod and container health

Pod status (Running, Pending, Failed).

Restart count.

Resource consumption (CPU, memory, disk).

Container start time, crash count, logs.

How metrics are exposed

User‑defined pod metrics : Applications expose Prometheus‑format metrics via an HTTP /metrics endpoint using client libraries (e.g., Go, Java, Python). Example Go code: package main import ( "net/http" "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/promhttp" ) var ( requestsTotal = prometheus.NewCounter(prometheus.CounterOpts{Name: "myapp_requests_total", Help: "Total number of requests."}) activeUsers = prometheus.NewGauge(prometheus.GaugeOpts{Name: "myapp_active_users", Help: "Number of active users."}) ) func init() { prometheus.MustRegister(requestsTotal) prometheus.MustRegister(activeUsers) } func handleRequest(w http.ResponseWriter, r *http.Request) { requestsTotal.Inc() w.Write([]byte("Hello, World!")) } func userLogin() { activeUsers.Inc() } func userLogout() { activeUsers.Dec() } func main() { http.HandleFunc("/", handleRequest) http.Handle("/metrics", promhttp.Handler()) http.ListenAndServe(":8080", nil) }

Kubernetes core component metrics : API Server, kubelet, and other components expose Prometheus metrics at /metrics (kubelet on port 10250 with HTTPS). Example curl command: curl -k -H "Authorization: Bearer ${TOKEN}" https://{node-ip}:10250/metrics

Exporter metrics : Exporters translate internal component state into Prometheus format. Common exporters include: kube‑state‑metrics : exposes the state of Pods, Deployments, Nodes, etc. node‑exporter : provides node‑level hardware and OS metrics (CPU, memory, disk I/O, network).

Collecting kubelet metrics with Prometheus

scrape_configs:
  - job_name: kubelet-metrics
    honor_timestamps: true
    metrics_path: /metrics
    scheme: https
    kubernetes_sd_configs:
      - role: node
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      insecure_skip_verify: true

Alternatively, use the API‑server proxy to scrape kubelet metrics:

scrape_configs:
  - job_name: kubelet
    honor_timestamps: true
    metrics_path: /metrics
    scheme: https
    kubernetes_sd_configs:
      - role: node
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      insecure_skip_verify: true
    relabel_configs:
      - source_labels: [__meta_kubernetes_node_name]
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics
        action: replace

Conclusion

Monitoring Kubernetes requires a layered approach: start with core component health, then observe node resources, followed by pod/container metrics, and finally network/service performance. Combining custom application metrics, core component metrics, and exporter data gives a complete observability picture, enabling rapid fault detection, performance tuning, and reliable operation.

For a turnkey solution, Tencent Cloud Observability Platform (TCOP) offers managed Prometheus, Grafana, APM, and a suite of exporters, allowing you to monitor Kubernetes clusters with minimal setup and integrated alerting.