Automatic Anomaly Detection for Server Failures Using DBSCAN at Netflix

In the early hours of the morning, the Netflix technical operations team was still investigating an outage and eventually traced the problem to a single misbehaving server hidden among tens of thousands.

Inspired by the heightened senses of the blind hero Daredevil, the team developed a system that can detect minute differences between servers, which may indicate hidden faults.

This article introduces the automatic anomaly‑detection technique and the repair of the problematic server.

Netflix runs tens of thousands of servers, with a failure rate typically below 1%; for example, a network glitch on one server can cause user latency that is not captured by standard health checks.

Such partially failing servers are harder to spot than completely downed ones, yet they degrade user experience and generate customer complaints.

The colored lines in the chart represent error rates of individual servers; the purple line stays consistently higher, suggesting an abnormal server that could be detected automatically from the time‑series data.

A simple threshold‑based alarm works only for clearly high error rates and suffers from frequent spikes, making it difficult to set a stable threshold for many servers.

To overcome these issues, the team applied clustering analysis, specifically the Density‑Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm, which groups similar data points without requiring labeled data.

DBSCAN Algorithm Overview

Proposed in 1996 by Martin Ester, Hans‑Peter Kriegel, Jörg Sander, and Xu Xiaowei, DBSCAN scans all data points and groups densely reachable points into clusters, using a distance threshold (Eps) and a minimum number of points (MinPts) to define neighborhoods.

Using DBSCAN to Identify Anomalous Servers

First, a metric such as error rate is selected, then a period of time‑series data is collected and processed with DBSCAN to flag servers that deviate from normal behavior; an example of the collected data is shown in the pink‑highlighted chart.

In addition to the metric, a minimum duration for an anomaly must be defined; once detected, the system triggers actions such as emailing or calling the responsible engineer, taking the server offline without stopping it, collecting diagnostic data, and pausing the server until a replacement is provisioned.

Send email or call the owner

Take the server offline but keep it running

Collect server data for further investigation

Stop the server and wait for the scaling system to replace it

Parameter Selection

DBSCAN requires two parameters, Eps (neighborhood radius) and MinPts (minimum points to form a cluster). In this deployment, the parameters are inferred via a simulated‑annealing optimization based on the observed number of anomalous servers, simplifying configuration for multiple Netflix teams.

To evaluate the system, a week of production data was collected and the algorithm’s detections were compared with manually identified anomalies; the results are illustrated in the following chart.

The evaluation shows that while the detection system is not 100 % accurate, its performance is satisfactory for Netflix’s operational needs, where occasional false positives are acceptable because the scaling system can quickly replace a mistakenly taken‑offline server.

Current detection relies on batch processing of collected data; shorter windows may introduce noise, while longer windows delay detection. Future improvements could involve real‑time stream processing frameworks such as Mantis or Apache Spark Streaming, as well as advances in online machine‑learning and data‑mining research.

Parameter tuning could also be enhanced by labeling data and training supervised models, which would outperform the current inverse‑optimization approach and allow the model to adapt to evolving workloads.

Conclusion

As Netflix’s infrastructure grows, automating operational decisions—such as automatically stopping problematic servers—improves availability and reduces the burden on reliability engineers; the anomaly‑detection system described here is one example, and many other automation opportunities remain to be explored.

Disclaimer: The content originates from public internet sources; the author remains neutral and provides it for reference and discussion only. Copyright belongs to the original authors or institutions; please contact for removal if infringed.