How QTSDB Turns InfluxDB into a Scalable Distributed Time‑Series Database

QTSDB Overview

Existing open‑source time‑series database InfluxDB only supports single‑node operation, which leads to slow queries, high machine load, and capacity limits under massive write workloads. To solve these problems, the 360 infrastructure team developed a clustered version called QTSDB based on InfluxDB 1.7.

Key Features

High‑performance storage specially designed for time‑series data, balancing write speed and disk usage.

SQL‑like query language with support for many aggregation functions.

Automatic cleanup of expired data.

Built‑in continuous queries for automatic aggregation.

Implemented in Go with no external dependencies, simplifying deployment and operation.

Dynamic horizontal scaling of nodes to store massive data volumes.

Replica redundancy and automatic failover for high availability.

Optimized data ingestion supporting high throughput.

Logical Storage Hierarchy

In InfluxDB, the top‑level entity is a database . Under each database, one or more retention policies define data lifetimes. Each retention policy is divided into shard groups (time‑based partitions); when a shard group expires, the whole group is deleted. Within a shard group, data is stored in multiple shards , each mapped to a physical node and backed by a TSM storage engine.

Cluster Structure

QTSDB consists of three parts: proxy , meta cluster , and data cluster . The proxy is stateless, receives client requests, and can be horizontally scaled behind LVS. The meta cluster stores the logical storage hierarchy and the mapping between shards and physical nodes, guaranteeing strong consistency via the Raft protocol; its metadata resides in memory with logs and snapshots persisted to disk. The data cluster holds the actual data; each shard corresponds to a TSM engine.

Data Access Flow

When a request arrives, the proxy queries the meta cluster to locate the appropriate database, retention policy, and shard group based on the request’s time range. For writes, the series key is hashed to select a specific shard; the write is performed on all replicas using a leader‑less multi‑write strategy. For reads, because the series key is unknown, the proxy must query every shard in the relevant shard groups, selecting an available replica for each shard. The number of shards per shard group equals the number of physical nodes divided by the replica count, ensuring balanced distribution.

Query Processing

InfluxDB’s SQL‑like queries are translated into storage keys. A reverse index (TSI) maps tag values to series keys; the query engine builds a set of series keys that satisfy the WHERE clause, concatenates the selected field keys, and retrieves the corresponding columns from the TSM engine. Non‑aggregated queries combine columns from the same series key based on timestamps, while aggregated queries first retrieve the relevant series keys, then apply the aggregation functions according to the GROUP BY clause.

Fault Handling

Writes are sent to all shard replicas; the meta cluster monitors data nodes via heartbeats and selects an online replica for reads. If a data node becomes temporarily unavailable, the proxy buffers writes locally and forwards them once the node recovers. For short‑term outages, the node can rejoin the cluster with its original identity; for long‑term failures, the node can be removed and later added as a new member, effectively expanding the cluster.

Data Cluster Scaling

When new data nodes are added, existing data is not automatically migrated. Instead, the current shard group’s end time is set, and a new shard group is created based on the updated node count, allowing new writes to be evenly distributed across all nodes without affecting reads of historical data.

Summary

QTSDB implements a clustered time‑series database where writes are directed to shards based on series keys, while reads require scanning all relevant shards. The system performs two‑stage aggregation—first within data nodes, then at the proxy—to produce the final result set. Although functional, QTSDB’s cluster features are still evolving and will be refined in future releases.