How to Build an Automated Fault‑Healing System for Enterprise Ops

1. Basic Fault‑Self‑Healing Process

Automation means extracting human expertise and codifying it into programs, similar to industrial or internet revolutions. For example, when a disk alert occurs, an operator would normally log into the server and clean the disk.

Next, we decompose the logic.

1.1 Abstracting the Alert‑Handling Flow

1) Pull disk alerts

2) Write a script or job to clean the disk

3) Design a module that connects the pulled alerts with the script execution.

1.2 Using CMDB for Resource Normalization

Different modules need different disk‑cleaning strategies. Introducing a CMDB (mapping devices, people, and services) allows us to translate an IP into a module, ensuring the correct cleaning plan is applied at the access, logic, and storage layers.

1.3 Integrating Enterprise Internal Gateways

If self‑healing fails, users must be notified. Besides invoking jobs, the system may need to call internal gateways for actions such as server restart or resource provisioning. Using a PaaS‑level ESB to wrap these gateways provides permission checks, rate limiting, statistics, routing, and self‑service access, avoiding direct calls to raw interfaces.

1.4 Connecting to Internal Monitoring Systems

Examples with Zabbix and Open‑Falcon illustrate how to pull alerts and push them to the self‑healing engine.

1.4.1 Zabbix Integration

The article "When Zabbix Meets Self‑Healing" describes pulling Zabbix alerts via ActionScript, invoking scripts, and pushing alerts to the self‑healing module for real‑time processing.

1.4.2 Open‑Falcon Integration

Open‑Falcon provides a callback feature, simplifying the flow. After receiving a callback, the system parses the fields. If the CMDB identifies hosts by IP while Open‑Falcon reports by endpoint (hostname), the CMDB’s auto‑discovery can map hostnames back to IPs.

Below is a self‑healing example for an Nginx disk alert: the system matches the alert to an Nginx‑specific cleaning package, removes log files, and completes the process in under 30 seconds.

2. The Two‑Sided Nature of Automated Fault Handling

Automatic fault handling is a double‑edged sword: if false alerts are processed automatically, it can cause serious damage (e.g., rebooting healthy servers due to network‑induced ping alerts). To mitigate this, a convergence module can aggregate alerts within a time window and require human approval for bulk actions.

For example, if Y alerts occur within X minutes, trigger an approval call. If the same host generates multiple alerts of the same type, subsequent alerts within the convergence window are ignored.

3. Handling Complex Alerts – Composite Packages

For multi‑step fault replacement scenarios, a binary‑tree structure can model success and failure branches, enabling orchestration of actions such as verifying a critical module, allocating a standby machine, and handling exceptions.

The diagram illustrates a “combo‑package” self‑healing solution, where atomic actions are assembled to meet diverse requirements, aligning with the concept of resource orchestration.

4. Technical Architecture of Fault Self‑Healing

Combining the basic workflow, the two‑sided nature, and complex handling yields the following architecture:

This architecture provides a practical reference for building enterprise‑level automated fault‑handling solutions.

5. Closing Thoughts

When AIOps becomes mainstream, it is essential to prioritize core reliability over flashy, over‑engineered designs. Follow a product‑roadmap approach: first ensure availability, then improve experience, and finally address scalability and ecosystem integration.