Why Prometheus’s TSDB Makes Monitoring Scalable: A Deep Dive

Background

Many beginners feel overwhelmed by Prometheus because it introduces a large number of concepts such as Instance, Job, Metric, Metric Name, Metric Label, Metric Value, Metric Type (Counter, Gauge, Histogram, Summary), DataType (Instant Vector, Range Vector, Scalar, String), Operator, and Function.

Concepts: Instance, Job, Metric, Metric Name, Metric Label, Metric Value, Metric Type (Counter, Gauge, Histogram, Summary), DataType (Instant Vector, Range Vector, Scalar, String), Operator, Function

As one instructor noted, Alibaba is a data company rather than a retail company; similarly, Prometheus is fundamentally a data‑driven monitoring system.

Daily Monitoring

To monitor each API of a web server (e.g., WebServerA), typical dimensions include service name (job), instance IP (instance), API name (handler), HTTP method (method), response code (code), and request count (value).

Example SQL‑like queries on the

http_requests_total

metric:

<code>SELECT * FROM http_requests_total WHERE code="200" AND method="put" AND created_at BETWEEN 1495435700 AND 1495435710;</code>

<code>SELECT * FROM http_requests_total WHERE handler="prometheus" AND method="post" AND created_at BETWEEN 1495435700 AND 1495435710;</code>

<code>SELECT * FROM http_requests_total WHERE handler="query" AND instance="10.59.8.110" AND created_at BETWEEN 1495435700 AND 1495435710;</code>

When monitoring 100 services, each with 10 instances, 20 APIs, 4 methods, collecting data every 30 seconds and retaining 60 days, the total rows exceed 13.8 billion, which is impractical for relational databases like MySQL. Therefore, Prometheus adopts a TSDB storage engine.

Storage Engine

TSDB perfectly matches the characteristics of monitoring data:

Massive data volume

Predominantly write‑heavy workload

Writes are almost sequential, ordered by time

Writes rarely modify old data; data is written shortly after collection

Deletes are block‑based, targeting whole time ranges

Data size typically exceeds memory; caching offers little benefit

Reads are ordered (ascending or descending) scans

High‑concurrency reads are common

TSDB stores each sample as two parts:

labels

(the dimensions that uniquely identify a time series) and

samples

(timestamp and value).

<code>{
  "labels": [{"latency": "500"}],
  "samples": [{"timestamp": 1473305798, "value": 0.9}]
}</code>

Internally, TSDB builds three indexes to accelerate queries:

Series Index : stores the ordered list of label‑value pairs for each series.

Label Index : maps each label to a list of its possible values and points to the corresponding series.

Time Index : maps time ranges to data blocks, allowing fast skipping of irrelevant chunks.

Data Computation

The powerful storage engine enables Prometheus to perform matrix operations on metric series using built‑in operators and functions, turning the system into a combined “data warehouse + computation platform” for monitoring.

One Calculation, Multiple Queries

Because the computation can be resource‑intensive, Prometheus supports recording rules that pre‑compute frequently used or expensive expressions and store the results as new time series, enabling fast repeated queries for dashboards and alerting rules.