Mastering Fault Self-Healing: Automate Disk Alerts and Scale Operations

Background

Nightly disk‑space alerts (available space below 20%) often wake engineers at 23:00 or later, forcing manual handling. These alerts are just the tip of the iceberg; relying on luck for small issues is risky.

Typical manual solutions include setting cron jobs on alert machines or writing scripts to compress logs and clean disk space. While feasible for a few servers, managing thousands with chaotic directory structures would require thousands of scripts and tasks.

Therefore, fault self‑healing becomes essential.

Fault Self‑Healing

Traditional incident response involves receiving an alert, logging into a jump host, manually fixing the issue, and restoring service. Fault self‑healing accepts alerts from the monitoring platform, matches them to predefined remediation workflows, and automatically restores service through automation.

To adopt self‑healing broadly, several prerequisites must be established.

1. Prerequisites

Directory Management Standards : A standardized directory structure enables a single set of automation scripts to manage all file resources.

Application Standards : Consistent application conventions allow automation scripts to manage all applications uniformly.

Monitoring Alert Standards : Standardized alerts ensure both the operations team and the self‑healing platform can quickly locate problems.

Standard Fault‑Handling Process : A defined process speeds resolution and builds a knowledge base for the operations team.

2. Monitoring Platform

The monitoring platform must provide fast, accurate fault location across multiple dimensions:

Hardware Monitoring : Primarily auxiliary, helping detect issues early.

Basic Monitoring : Tracks CPU, memory, disk usage; can report top‑resource‑consuming processes and custom disk‑cleanup policies.

Application Monitoring : Monitors health checks, ports, and custom alerts; can trigger application restarts.

Middleware Monitoring : Observes cluster health (e.g., Eureka instances, RabbitMQ nodes, Redis clusters) and can act on individual node failures.

Additional dimensions can be added as operational experience grows.

3. Fault‑Self‑Healing Platform

(1) Multi‑Alert Sources

The platform must support multiple monitoring tools (Zabbix, Nagios, Open Falcon, Prometheus, etc.) and REST APIs to remain adaptable to evolving monitoring ecosystems.

(2) Unified Data Source

A CMDB serves as the authoritative source, providing configuration data for monitoring, self‑healing, and other upper‑layer applications.

In the ITIL framework, CMDB is the foundation for building other processes, offering configuration data services that map relationships between applications and ensure data accuracy and consistency.

Implementing CMDB involves challenges such as gaining internal team acceptance, defining responsibility boundaries, establishing management standards, organizing resources by department and business, and integrating physical machines, virtual machines, network devices, databases, and middleware.

(3) Fault Handling

Remote script execution is essential. Common tools include Ansible, SaltStack, or a central control machine executing commands via SSH. More advanced or elegant methods may also be employed.

(4) Result Notification

Regardless of success, the outcome must be communicated for possible human intervention. Notification channels can include email, WeChat, DingTalk, SMS, phone calls, and others.

Conclusion

Fault self‑healing can resolve many issues but remains only one part of the incident‑resolution workflow. It must be tightly integrated with other components and coordinated by operations personnel to avoid unintended triggers during routine maintenance.

Ultimately, self‑healing is a tool; its broader adoption depends on diligent, hands‑on practice by the operations team.