Ant Financial Data Governance: Practices and Challenges in Data Quality Management

Earlier this year, Ant Financial's ATEC City Summit was held in Shanghai. In the financial intelligence special forum, senior data technology expert Li Junhua from Ant Financial Data Platform Department gave a presentation titled "Ant Financial Data Governance: Data Quality Governance Practice".

In the talk, Li introduced Ant Financial's data architecture system's immune system — the data quality governance system, focusing on data quality implementation and Ant's data quality governance practice and challenges.

Senior data technology expert Li Junhua

1. Overview of Data Governance

In recent years, Ant Financial has continuously upgraded its data architecture to solve data physical island problems. Now the foundation of Ant and the entire Alibaba Group is unified on the same platform, reducing the overall threshold for one‑stop development and enabling all engineers to easily work with data. The current data architecture resolves physical data islands, while the governance system now focuses on logical islands.

Before discussing data governance, let's talk about data value.

Previously, data needed to be processed by a dedicated team to delete valueless data and handle data onboarding/offboarding. Judging data value was difficult; most data would be onboarded but not offboarded, leading to accumulation of valueless data. Now Ant cares about offboarding valueless data and maximizes data asset value.

Ant has a complete data asset grading and usability model, driving full utilization of data assets to create more value. However, if data is used but of low quality, the asset value is greatly reduced.

Data Quality Generation Analysis

Next, we will introduce Ant Financial's practice ideas and solutions in data quality governance, sharing two cases. The diagram below shows the full process of abstract data extraction.

When a business colleague enters data with a small error, it can cause data quality issues, such as wrong industry information or a typo.

In traditional database asset development, data is produced at the source, processed, and sent out, following "from business to business". The new solution differs: after processing, data is returned to the data system. For example, in Sesame Credit score calculation, many unseen scenarios involve data returning to the system, and each step may have quality issues.

2. Challenges of Data Quality Governance

The left side of the diagram below shows Ant's business forms.

Now Ant's business scenarios are not limited to statistical analysis; Sesame Credit, Huabei, Jiebei, and "310" lending are all driven by data. Ant's business model is a fusion of technology, data, and algorithms to maximize value. Data quality governance faces many challenges from business, data, and user aspects.

3. Data Quality Governance Practice

Data Quality Governance Approach

Financial business colleagues feel that the lifecycle of internet finance has shortened and changes rapidly, much faster than traditional banking.

Currently, both Ant Financial and Alibaba talk about "data businessization, business dataization", with data and business developing together, entering a deep‑water stage. Previously, Ant's business was T+1, but the old architecture cannot support future high‑timeliness demands. Ant's data volume is large, and data business drives talent upgrades. Besides data algorithm R&D, other technical staff also use data on the platform, with varying data understanding, making data quality assurance crucial.

How to achieve data quality governance?

First, a clear organization is needed, forming the soil for continuous corporate culture. Data quality governance culture must be determined, organized, and sustained. Based on organizational assurance and quality culture, Ant emphasizes both development flow and data flow. In finance, development flow control is stricter. For internet finance, strong control is needed because business forms dictate short development cycles. Ant implements strong control in a one‑stop data development platform with hierarchical control. After demand submission, it is graded, labeled, and enters different processes. Development flow also uses hierarchical control, defining levels on the same standard to level the flows. For data flow, after an application is released to production, most effort focuses on data flow: daily data collection from production to processing platform, algorithm computation, and returning data to production, forming a closed loop.

Now Ant has built many capabilities in the data flow link. For data flow, if the source is polluted and cannot control downstream contamination, repair costs increase downstream.

Based on the above data quality governance ideas, Ant monitors the entire system during data platform operation; if a data quality fault occurs, it can be repaired promptly.

Moreover, from development to production, Ant has done extensive work because many data development colleagues use the platform, aiming to lower usage thresholds. For the full data flow, four major capabilities have been built: perception, identification, intelligent healing, and operation.

The platform must perceive task faults and data quality issues, identify potential risks to understand compromised data timely. After risk identification, intelligent healing uses AI algorithms to assist data processing, combining perception with algorithmic ability to self‑heal data infections.

Finally, operational capability: if data quality is good, it can be invisible to users, who need not worry about data usability. Data quality is not only a development or business issue but requires all participants to solve together, which is the governance thinking.

Ant Data Quality Governance Architecture

The diagram below shows Ant Financial's data quality governance architecture system.

At the system layer, following the discussed ideas, the development stage focuses on data testing, release control, and change management. Change issues involve system‑level change management and online system interconnections. Online data source changes can affect data operations and cause quality problems.

The online development part provides interfaces for the data operation system to notify users of changes that may affect data operations. For release control, Ant has invested heavily. Currently, Ant has no dedicated data testing staff; all are full‑stack engineers, so control may not be strong, but they have robust release control, executing checks related to experience, standards, performance, and quality.

In the production stage, focus is on quality monitoring, emergency drills, and quality governance. Quality monitoring alarm system is like a car brake, essential. Ant also conducts data attack‑defense drills, where engineers create faults to test system detection and repair. In quality governance, applications are inspected regularly after release based on level, analyzing impact on data quality. Overall, the system layer combines original data with machine learning to auto‑configure strategies.

Data Quality Governance Solution

The diagram below shows Ant Financial's pre‑, during‑, and post‑ data quality solutions.

Overall, the pre‑stage includes demand, development, and pre‑release phases; Ant can achieve controllable, simulatable, and gray‑scale. The during‑stage focuses on monitoring; problems are not scary but need autonomous detection. Ant also implements proactive attack drills to discover weaknesses. Additionally, strong emergency capability triggers plans to turn uncertain data risks into certain ones. The post‑stage emphasizes auditing and measurement via effective indicators and controls to identify and continuously improve weak links.

Data Quality Governance Cases

Finally, two cases of Ant Financial's data quality governance:

Case 1: Under Ant's data governance architecture, the release stage implements a strong control process. Any script must pass checks before submission, then be released online, and undergo another check.

Case 2: Data governance covers the whole link; for different data versions, data collection moves data from one end to another without processing, allowing intentional fault injection to test the governance system's detection and response. Data processing has another architecture, involving logical processing and fault injection considerations. Ant has invested heavily in attack‑defense drills for data quality governance.

Click the lower left "Read original" to go to Ant Financial's official website for more.