Can AI Predict Disk Failures? RGF + Transfer Learning for Reliable Data Centers

IBM Research presented "Predicting Disk Replacement towards Reliable Data Centers" at KDD 2016, highlighting that disks are the most common and failure‑prone hardware in modern data centers.

Despite RAID protection, system availability suffers; traditional SMART‑based models lack robust attribute selection, accuracy, and reusability.

The paper proposes an automatic, precise disk‑failure prediction method that decides whether a disk should be replaced soon, illustrated by two diagrams comparing traditional anomaly detection with proactive prediction.

Challenges of Disk Failure Prediction

Not all SMART attributes relate to failures – selecting relevant attributes is essential.

Highly imbalanced failure data – only ~2% of disks are replaced, making minority class detection difficult.

SMART variations across manufacturers – models differ, requiring adaptable prediction methods.

Design Idea

The solution consists of five steps:

Select SMART attributes using changepoint detection to identify attributes correlated with disk replacement.

Generate time series by applying exponential smoothing to create informative sequences.

Address data imbalance through down‑sampling of healthy disks via K‑means clustering to balance classes.

Classify disk state with the Regularized Greedy Forest (RGF) algorithm, which classifies each time series as healthy (0) or failing (1).

Transfer learning to adapt models trained on one disk model to other models from the same manufacturer, mitigating sample selection bias.

1. Selecting SMART Attributes

Changepoint detection identifies permanent spikes in SMART metrics (e.g., SMART_187_raw) that indicate impending failure.

2. Generating Time Series

Exponential smoothing (S_t = α·Y_t + (1‑α)·S_{t‑1}) retains historical information while emphasizing recent data, enabling early fault prediction.

3. Solving Data Imbalance

Healthy disk series are clustered with K‑means; the nearest points to each centroid are selected to represent the majority class, achieving a balanced dataset.

4. Disk State Classification

RGF improves on GBDT by globally optimizing the greedy forest, adding regularization to prevent over‑fitting.

5. Transfer Learning

Domain adaptation aligns feature distributions between source (labeled) and target (unlabeled) disk models, allowing a model trained on one model to predict failures on another.

Conclusion

The study presents a fully automated, accurate disk‑failure prediction pipeline that selects relevant SMART attributes, creates smoothed time series, balances training data, classifies disk health with RGF, and applies transfer learning across models, achieving high precision and recall while reducing the number of required models.