How Nightingale Transforms Monitoring for Scalable Stability

Night­ingale Design and Product Overview

Night­ingale (Nightingale) is Didi's open‑source monitoring project derived from the commercial ECMC platform. It was created to address the limitations of legacy systems like Zabbix and early Open‑Falcon, especially the need for high‑capacity, high‑performance time‑series storage and flexible alerting.

The evolution path moved from early InfluxDB‑based solutions to ODIN monitoring, which handled billions of metrics per day, and finally to Night­ingale, an upgraded Open‑Falcon with many architectural refinements.

Key Improvements Over Open‑Falcon

Alert Engine Refactor : switched from pure push to a push‑pull hybrid, added missing‑data alerts, multi‑condition alerts, and production‑grade features such as alert convergence, claim, and escalation.

Service Tree Integration : introduced a hierarchical service tree (navigation object tree) allowing policies to inherit to child nodes, simplifying configuration.

Index Module Upgrade : replaced MySQL‑based index storage with an in‑memory index to handle billions of metric entries.

Time‑Series Storage Optimization : adopted Facebook’s Gorilla compression and in‑memory storage for recent data, dramatically improving query speed.

High‑Availability Alert Engine : heartbeat‑driven automatic removal of failed judge instances and similar HA for the index module.

Built‑in Log Monitoring : native log matching and metric extraction, reducing the need for intrusive instrumentation.

Operational Simplicity : merged multiple modules, enabling internal method calls for better performance.

Centralized Configuration : extracted common settings into shared config files with sensible defaults.

Night­ingale retains the original data model (metric+tag) and adds an extra field for unstructured information such as trace IDs or error logs.

System Architecture

The collector (agent) runs on target machines, gathering metrics, logs, and supporting plugins. Data is pushed via long‑lived connections to

transfer

, a stateless, horizontally‑scalable component that hashes data to appropriate

tsdb

shards. Each

tsdb

stores data in memory using Gorilla compression and persists with rrdtool, while also feeding an

index

module for fast look‑ups.

The

judge

module pulls alert policies from

monapi

, evaluates metrics, and emits alerts to a Redis queue.

monapi

consumes these alerts, provides a web API for front‑ends, and forwards queries to

index

and

transfer

. All backend services include heartbeat mechanisms for automatic failover, ensuring high availability.

Alert Engine Features

Night­ingale offers production‑grade alerting: multi‑level severity (P1‑P3) with different notification channels, alert convergence, callbacks for automated remediation, claim and escalation workflows, time‑window policies, silent recovery, inheritance via the service tree, AND‑condition alerts, tag inclusion/exclusion filters, and memory‑efficient caching.

Event Handling

All alert events and active incidents are stored for post‑mortem analysis. Alerts are placed in a Redis queue; external sender modules (email, DingTalk, WeChat) consume them. Callbacks enable integration with internal automation for self‑healing scripts, which Didi runs thousands of times weekly.

Future Directions

Planned enhancements include an aggregation module for cluster‑level metrics, tighter integration with cloud‑native ecosystems (automatic Kubernetes and cAdvisor metric collection), and community‑driven maintenance of legacy Open‑Falcon plugins.

How Night­ingale Supports the Stability Lifecycle

Fault Prevention

Provides APIs to audit strategy coverage, detect orphaned alerts, and compute a “monitoring health score.” Quantifies risk via callback coverage and alarm statistics, enabling proactive risk reduction.

Fault Detection

Rich, real‑time metrics and flexible alert policies ensure rapid detection of anomalies across services.

Fault Localization

Dashboard thresholds and drill‑down links let operators trace high‑level business anomalies down to module‑level metrics; integration of alert and change events aids root‑cause analysis.

Fault Mitigation

Alert callbacks trigger automated remediation scripts (e.g., log cleanup) to achieve self‑healing without human intervention.

Post‑mortem Review

Comprehensive alert histories support detailed incident reviews and continuous improvement of on‑call processes.