Building a Complete Linux Monitoring Dashboard with Prometheus, Pushgateway, and Grafana

This article guides Linux system administrators and DevOps engineers through creating a full-featured, customizable monitoring dashboard for one or many instances using Prometheus, Pushgateway, and Grafana.

It begins by explaining why a dedicated monitoring solution is needed, then outlines the architecture: a Bash script periodically pushes process metrics to Pushgateway, Prometheus scrapes Pushgateway and stores the time‑series data, and Grafana visualizes the data.

Installation commands for each component are provided, for example: wget https://github.com/prometheus/pushgateway/releases/download/v0.8.0/pushgateway-0.8.0.linux-amd64.tar.gz tar xvzf pushgateway-0.8.0.linux-amd64.tar.gz cd pushgateway-0.8.0.linux-amd64/ && ./pushgateway & wget https://github.com/prometheus/prometheus/releases/download/v2.9.2/prometheus-2.9.2.linux-amd64.tar.gz tar xvzf prometheus-2.9.2.linux-amd64.tar.gz cd prometheus-2.9.2.linux-amd64/ wget https://dl.grafana.com/oss/release/grafana_6.2.0-beta1_amd64.deb && sudo dpkg -i grafana_6.2.0-beta1_amd64.deb

A sample Bash script ( #!/bin/bash z=$(ps aux) while read -r line; do var=$var$(awk '{print "cpu_usage{process=\""$11"\", pid=\""$2"\"}", "$3"}'); done <<< "$z" curl -X POST -H "Content-Type: text/plain" --data "$var" http://localhost:9091/metrics/job/top/instance/machine ) collects process information via ps aux , formats it as Prometheus exposition strings, and posts it to Pushgateway.

Prometheus is configured with a 1‑second scrape interval and static targets for both the Prometheus server (port 9090) and Pushgateway (port 9091). Grafana is then connected as a data source, and several panels are built: a circular gauge, horizontal and vertical bar gauges, and a time‑series line chart. The panels use PromQL queries such as sum(cpu_usage) for total CPU usage and avg(cpu_usage) for average usage, as well as topk(10, cpu_usage) to show the ten most resource‑intensive processes.

The article also demonstrates how to add ad‑hoc filters in Grafana for dynamic querying, allowing users to focus on specific processes or metrics without modifying the underlying queries.

Finally, a concise recap reinforces the key steps: installing the stack, writing the metric‑export script, configuring Prometheus, building Grafana dashboards, and optionally using ad‑hoc filters for deeper analysis.