How Intelligent Ops Transforms Monitoring: Multi‑Dimensional Anomaly Detection & Smart Alert Merging

Background

Intelligent operations find a natural fit in monitoring because the field generates massive data that can be analyzed and processed intelligently.

In monitoring, we face many needs such as anomaly detection for large numbers of metrics, each requiring different detection methods.

We also receive a huge volume of alerts that need to be merged and distilled to highlight the most important information, identifying root causes and downstream effects.

1. Multi‑Dimensional Anomaly Detection

Anomaly detection is crucial in operations; real‑time, accurate detection enables timely actions to minimize fault impact.

When systems become large and complex, finding anomalies among hundreds of monitoring strategies is difficult, especially with static thresholds.

Static thresholds work well for simple host metrics like CPU and memory, where a fixed percentage (e.g., 60%) can trigger alerts, but they struggle with business‑level metrics that vary with traffic.

Statistical methods and machine‑learning models (e.g., classification models) are introduced to handle dynamic, periodic, and high‑variance metrics.

We use LightGBM with selected features to build models that label normal versus abnormal points, visualizing results with three severity levels: normal (yellow), severe (red), and abrupt (critical).

2. Intelligent Alert Merging

When a system experiences an incident, a flood of alerts overwhelms operators; intelligent merging reduces noise and extracts actionable information.

We propose merging alerts based on multiple dimensions (cluster, IP, service, time window) and calculate information purity (e.g., Gini impurity) to choose the best merging dimension.

The process builds a tree where alerts are grouped hierarchically; each node represents a merge dimension, and leaf nodes contain the final consolidated alerts.

Examples show how 22 individual host‑down alerts can be merged into a single alert that still provides the proportion of affected hosts and a link to detailed per‑host information.

We also merge based on alarm type, combining related alerts across clusters, services, or network segments to help operators quickly pinpoint the root cause.

3. Knowledge Graph Construction

Building a knowledge graph is essential for a “smart ops brain,” enabling root‑cause analysis by linking configuration management (CMDB), monitoring, management, and cloud platforms.

We extract entities such as clusters, servers, ports, and processes, and discover relationships like association, causality, and deployment.

The graph captures call‑chain information, layered monitoring metrics (server, system, business), and infrastructure dependencies (e.g., DNS).

Features such as department ownership, usage patterns, and traffic trends are encoded, and an isolation‑forest‑based decision model maps these features to unified representations.

Correlation mining (using algorithms and Pearson coefficients) validates causal links between metrics, producing clear association diagrams.

4. Intelligent Root‑Cause Analysis

Root‑cause analysis relies on abundant labeled fault data to train models, but real‑world fault data is scarce; therefore, we adopt a dynamic decision approach.

Real‑time anomalies and change events feed a root‑cause component that selects between state‑machine and behavior‑tree strategies, the latter offering better extensibility.

Behavior trees consist of logical and execution nodes; logical nodes guide the investigation sequence based on expert experience, while execution nodes perform data processing and metric correlation.

The framework integrates data extraction, metric and call‑chain correlation, change and hardware status, and uses curve similarity to verify strong correlations.

Illustrative cases show how a host failure propagates to virtual machines and causes high latency, or how traffic spikes increase packet loss, demonstrating the system’s ability to visualize and explain complex incidents.