Automated Fault Detection and Repair System for Grab's Data Pipelines (Hugo) – Architecture, Implementation, and Impact

At the upcoming DA Digital Intelligence Conference (July 25‑26, Shenzhen), Grab experts Dr. Chen Jia and platform lead Cheng Feng will share their experience with Data Mesh implementation, data‑development automation, and an AI‑driven automatic data‑analysis platform.

Problem Context In Grab's data ecosystem, the Hugo service manages more than 4,000 user‑created pipelines. System stability depends heavily on data source health and internal component performance, yet occasional pipeline failures are inevitable. When automatic retries fail, manual intervention is required, exposing several challenges: data teams cannot promptly detect anomalies, engineers are overwhelmed by a flood of ad‑hoc requests, and DPI reports often lack root‑cause analysis, leading to prolonged downtime and higher costs.

Solution Overview By analyzing historical failure patterns and evaluating remediation costs, Grab designed a comprehensive automation framework that integrates signal collection, intelligent diagnosis, and automatic remediation to reshape the fault‑management workflow.

01 Architecture Design: From Concept to Implementation

The design follows three core principles:

Precise fault‑pattern identification based on historical data and first‑principles.

Time‑series‑driven diagnosis that correlates pipeline execution steps.

Layered repair capability that prioritises high‑frequency scenarios and gradually expands to complex cases.

The system consists of six tightly coupled modules forming a closed loop from monitoring to action:

Signal Collection Module – continuously captures three key health indicators:

Fault‑callback signals via Airflow callbacks.

SLA‑alarm signals generated by Grab's Genchi data‑quality platform.

Data‑integrity signals that validate source‑target table consistency.

Diagnosis Engine Module – serves as the intelligent core, aggregating signals and internal platform data to perform root‑cause analysis at thousand‑level concurrency. For SLA violations, it sequentially triggers an Airflow analyzer, a resource‑monitor analyzer, and a dependency analyzer, reducing average diagnosis time from hours to minutes.

RCA Knowledge Base – stores structured fault features, owner mappings, and remediation steps, enabling automatic pattern matching for new incidents.

Automatic Repair Module – plugin‑based design supports rapid integration of custom processors. It applies exponential back‑off retries for transient errors (e.g., replica lag) and automatically creates tickets for persistent issues (e.g., schema conflicts), operating asynchronously to avoid impacting the main data path.

Data Health API – exposes standardized health metrics, allowing downstream platforms to subscribe to table‑level health status and retrieve detailed diagnostic reports on demand.

Health Dashboard – visualises system health as a heatmap, provides owner information, remediation guides, and repair progress, and offers an admin view with confidence scores and evidence chains.

02 Implementation Details: Technical Breakthroughs and Engineering Practices

Signal collection unifies heterogeneous sources: Airflow callbacks for fault signals, Genchi‑generated SLA and integrity signals, and custom Kafka‑based metrics. The diagnosis engine employs a fine‑grained concurrency model—thousands of parallel processes for fault signals and Kafka‑partition‑aligned parallelism for SLA/quality signals—ensuring high throughput without resource contention. The auto‑repair framework demonstrates elastic scaling in production.

Integration with the internal event‑management platform Kinabalu enables automatic Slack notifications, Jira ticket creation, and Splunk alert suppression, forming an end‑to‑end automated incident‑response pipeline.

03 Impact Evaluation: Data‑Driven Value Verification

Data visibility coverage rose from 68 % to 97 %; average issue‑detection latency dropped to 8 minutes.

72 % of common failures are now auto‑repaired, reducing manual interventions by 63 %.

On‑call team ticket volume decreased by 54 %; complex‑issue handling time fell by 41 %.

DPI report RCA completeness reached 100 %, improving breach‑handling efficiency by 2.8×.

Business impact includes a 22 % reduction in data‑preparation time for the payment‑risk team and a rise in input‑data availability for the logistics ETA model to 99.92 %. The framework has been reused by seven other platform teams, including ad‑serving and user‑profile systems.

04 Future Outlook: Building a New Paradigm for Intelligent Operations

Planned upgrades:

Intelligent repair enhancements using reinforcement‑learning‑driven retry interval adjustment and dynamic resource allocation to cut end‑to‑end latency by 30 %.

Dashboard extensions with health‑trend prediction (30‑minute early warning) and topology visualisation of pipeline‑resource dependencies.

Expanded diagnostic scope to incorporate Flink job health, Kafka consumer lag, and service‑mesh‑based call‑chain tracing for cross‑system root‑cause analysis.

Deeper ecosystem integration with SRE monitoring for automatic pipeline pausing on cluster anomalies, and CI/CD pipelines that convert frequent failure patterns into automated test cases.

05 Conclusion – The Evolution Philosophy of Automated Operations

The Hugo journey validates three core ideas:

Problem‑driven incremental innovation—starting from high‑frequency database‑connection timeouts and progressively tackling complex schema‑evolution scenarios.

Human‑machine collaborative knowledge—encoding engineers' troubleshooting experience into diagnostic rules and continuously refining them with machine‑learning feedback.

Platform‑level capability output—standardised APIs and a plug‑and‑play architecture that enable cross‑team reuse and turn each incident into reusable knowledge.

By systematically combining signal acquisition, intelligent diagnosis, and resilient remediation, Grab demonstrates that operations can shift from reactive “fire‑fighting” to proactive, predictive governance, creating a virtuous cycle of value creation.