Building a Prometheus‑Based Monitoring System with Docker and Kubernetes

1. Introduction

The author introduces the need for a monitoring system before starting a flash‑sale (秒杀) series, outlining three monitoring layers: API, hardware, and application.

2. Overview of Prometheus

What can Prometheus do?

S1: Periodically scrape metric data from many sources.

S2: Store the time‑series data in its own database.

S3: Query data in real time using PromQL.

Supports visual dashboards (often with Grafana).

Custom alert rules and notifications.

Dynamic service discovery.

Multi‑source monitoring.

2.1 Exporters

Node Exporter – hardware metrics for Linux/Unix.

Blackbox Exporter – network probing (HTTP, DNS, TCP, …).

Docker Daemon Exporter – container CPU, memory, network.

MySQL Exporter – MySQL performance and status.

Kubernetes State Metrics – cluster state (pods, nodes, deployments).

Nginx Exporter – Nginx request latency, connections.

Redis Exporter – Redis memory usage, command stats.

Elasticsearch Exporter – cluster health and index info.

More exporters are listed in the official documentation.

2.2 Summary of the workflow

Install exporters → Prometheus scrapes them → Grafana visualizes the data.

3. Detailed Implementation

3.1 Prometheus installation – Docker

Pull required images:

// Pull images
docker pull prom/node-exporter
docker pull google/cadvisor
docker pull prom/prometheus
docker pull grafana/grafana

// Run node‑exporter
docker run -d --name node-exporter --restart=always -p 9100:9100 \
  -v "/proc:/host/proc:ro" -v "/sys:/host/sys:ro" -v "/:/rootfs:ro" prom/node-exporter

// Run cAdvisor
docker run --privileged=true -v /:/rootfs:ro -v /var/run:/var/run:rw \
  -v /sys:/sys:ro -v /var/lib/docker/:/var/lib/docker:ro \
  -v /dev/disk/:/dev/disk:ro -p 8080:8080 -d --name=cadvisor \
  --restart=always google/cadvisor:latest

Deploy Prometheus container with a custom prometheus.yml configuration:

// Create config directory
mkdir -p /opt/prometheus && cd /opt/prometheus
vim prometheus.yml

// Example prometheus.yml
global:
  scrape_interval: 60s
  evaluation_interval: 60s
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
        labels:
          instance: prometheus
  - job_name: linux
    static_configs:
      - targets: ['localhost:9100']
        labels:
          instance: localhost
  - job_name: cadvisor
    static_configs:
      - targets: ['localhost:8080']
        labels:
          instance: cAdvisor

// Run Prometheus
docker run -d --name prometheus --restart=always -p 9090:9090 \
  -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

The configuration defines global scrape intervals and three jobs: Prometheus itself, the node exporter, and cAdvisor.

3.2 Prometheus installation – Binary

The author references an external blog for binary deployment and recommends using Docker for simplicity.

3.3 Grafana configuration

// Prepare storage
mkdir /opt/grafana-storage
chmod 777 -R /opt/grafana-storage

// Run Grafana container
docker run -d --name grafana --restart=always -p 3000:3000 \
  -v /opt/grafana-storage:/var/lib/grafana grafana/grafana

// Access Grafana at http://
:3000/

After starting Grafana, add a Prometheus data source, then import dashboards using Grafana’s template IDs.

4. Additional Features

Alertmanager

Manages alerts, supports silencing, grouping, and routing.

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

Changing storage backend

Prometheus can use remote write endpoints for long‑term storage.

--web.enable-remote-write-receiver /api/v1/write

The author notes they have not implemented remote storage in this guide.

5. Conclusion

After completing this setup, the flash‑sale series can use the visualized performance dashboards for load testing, and the author promises future articles on advanced Prometheus usage.