Comprehensive Introduction to Prometheus: Architecture, Metrics, Configuration, PromQL, Exporters, Visualization, and Alerting

Prometheus is an open‑source monitoring solution that collects metrics from instrumented services, stores them in a time‑series database, and provides powerful query and alerting capabilities.

Overall Architecture – Services (Jobs) expose metrics via HTTP endpoints or exporters; Prometheus pulls these metrics at regular intervals, stores them locally, and optionally forwards them to external systems.

Metric Exposure – Each monitored service is a Job . Metrics can be exported using the official client libraries or third‑party exporters (e.g., MySQL, Consul). Example of exposing metrics in Go:

package main
import (
    "net/http"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)
func main() {
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":8080", nil)
}

Scrape Configuration – Static and dynamic service discovery are supported. A simple static scrape config looks like:

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

Dynamic discovery can be done via Consul, Kubernetes, DNS, etc., by adding the appropriate *_sd_configs block.

Storage and Query – Metrics are stored as time series with a metric name, label set, timestamp, and value. Queries are performed using PromQL, which supports instant vectors, range vectors, and scalar values.

Metric Types – Prometheus defines four types: counter , gauge , histogram , and summary . Counters only increase, gauges can go up or down, histograms bucket observations, and summaries calculate quantiles on the client side.

PromQL Basics – Example of an instant query:

go_gc_duration_seconds_count

Range query for the last 5 minutes:

go_gc_duration_seconds_count[5m]

Rate calculation:

rate(http_requests_total[1m])

Instantaneous rate with irate and aggregation with sum by() or without() are also demonstrated.

Exporters and Custom Metrics – Custom exporters can be built using the client libraries. Example of a custom counter with labels:

myCounter := prometheus.NewCounterVec(
    prometheus.CounterOpts{Name: "my_counter_total", Help: "custom counter"},
    []string{"label1", "label2"})
myCounter.With(prometheus.Labels{"label1": "1", "label2": "2"}).Inc()

Histogram bucket configuration must match the expected value range to obtain accurate quantiles:

MyHistogram := prometheus.NewHistogram(prometheus.HistogramOpts{
    Name: "my_histogram_bucket",
    Help: "custom histogram",
    Buckets: []float64{0.1, 0.2, 0.3, 0.4, 0.5},
})
MyHistogram.Observe(0.3)
MyHistogram.Observe(0.4)

Dynamic Configuration Reload – Start Prometheus with --web.enable-lifecycle , then reload the config via HTTP POST:

prometheus --config.file=/etc/prometheus.yml --web.enable-lifecycle
curl -v -X POST http://localhost:9090/-/reload

The reload handler sends a signal through an internal channel to apply the new configuration without restarting.

Visualization with Grafana – Add Prometheus as a data source in Grafana, create dashboards, and write PromQL queries in panels to visualize metrics.

Alerting with Alertmanager – Define alert rules in alert_rules.yml , configure Alertmanager as the notification endpoint, and set up receivers (e.g., email). Example alert rule:

groups:
- name: simulator-alert-rule
  rules:
  - alert: HttpSimulatorDown
    expr: sum(up{job="http_srv"}) == 0
    for: 1m
    labels:
      severity: critical

Alertmanager configuration includes SMTP settings, routing, and silencing options.

Overall, the guide walks through the complete lifecycle of monitoring with Prometheus—from metric exposition and collection to querying, visualization, and alerting—providing practical code snippets and configuration examples.