MySQL Governance Practices at Industrial and Commercial Bank of China

Based on the 2020 DTCC Database Conference, Industrial and Commercial Bank of China (ICBC) has been promoting MySQL since 2014; today its production environment hosts nearly 10,000 MySQL nodes, with core high‑level applications increasingly relying on this database.

Speaker: Lin Zhenxi, a MySQL architect at ICBC since 2014, responsible for MySQL architecture design, development standards, and technical training.

The governance of MySQL in ICBC is divided into three parts: the current situation and challenges, the ideas and concrete solutions for addressing those challenges, and future improvement directions.

1. Current Situation and Challenges

From June 2019 to the present, the number of MySQL nodes has multiplied several times, with a large proportion serving core A‑level applications. The bank’s strategy of using MySQL for most new databases, combined with high‑capacity, high‑concurrency, and elastic‑scaling business demands, has driven this rapid growth. Operational pressure is high due to the sheer scale: monitoring, alerting, and fault recovery become difficult without automation. Traditional physical‑machine deployment wastes CPU resources, prompting a shift to containerized deployment (≈90% of nodes run in containers), which improves utilization by 4‑5×.

2. Governance Ideas and Solutions

The governance framework follows three stages: pre‑emptive prevention, emergency response, and post‑mortem diagnosis, all built on a foundation of strict standards.

2.1 Standardization Effects

Standards define table design rules (e.g., every table must have a primary key, no per‑table character set or collation, use InnoDB), quantitative limits (slow‑SQL scan‑rows to result‑rows ratio ≤ 100:1, transaction updates ≤ 100 000 rows), and bug avoidance (prohibit large‑table TRUNCATE and problematic REPLACE INTO).

2.2 Implementation Details

Table‑structure review is automated: new tables are generated by the review system, eliminating manual SQL errors; schema changes (adding fields or indexes) are also auto‑generated. Code review leverages MyBatis‑based scanning. Health checks use a dedicated MySQL view that records execution counts, total time, and scanned rows for each SQL; by sampling this view at two time points, the team identifies slow or resource‑heavy queries and automatically kills them using process‑list information. Large transactions (≥ 100 000 rows) are detected and automatically terminated to prevent binlog bottlenecks.

2.3 Three Main Workflows

Pre‑emptive Prevention: Table‑structure and code reviews enforce standards; automated slow‑SQL detection and killing reduce issues before they affect production.

Emergency Response: Real‑time monitoring captures performance anomalies; automatic killing of offending threads and segregation of OLTP and batch workloads mitigate impact; large‑transaction handling and failover mechanisms are in place.

Post‑mortem Diagnosis: Comprehensive data collection (high‑density thread metrics, low‑density slow‑SQL metrics) enables detailed root‑cause analysis after incidents.

3. Future Improvement Directions

Future work focuses on earlier problem detection (predictive analysis based on performance trends), rapid localization during incidents, and self‑healing automation (e.g., automatically creating missing indexes when full‑table scans are detected).

The speaker concluded that the presented material is introductory and welcomes further discussion and feedback.