How DataBus Enables Real-Time, Scalable Database Synchronization for Oracle Migration

1 Design Concept

DataBus originated from Lufax’s large‑scale Oracle de‑commissioning and architecture transformation projects. The goal is to replace Oracle not only at the database layer but also to refactor applications toward micro‑service architectures, resulting in many more source and target applications and databases.

During the split, data and applications that were previously co‑located in a single database become distributed, increasing the number of synchronization links and complexity.

DataBus addresses the need for fine‑grained horizontal data splitting and real‑time synchronization among multiple sources such as applications, databases, and big‑data storage engines.

Challenges of Oracle De‑commissioning

Complex large transactions and advanced Oracle features must be migrated to the application layer, and the explosion of application and database instances creates geometric growth in synchronization chains.

2 Architecture

DataBus Architecture

DataBus ensures second‑level synchronization latency and supports breakpoint‑resume and idempotent processing when components fail.

It guarantees data consistency between source and sink, preventing loss or format changes.

DataBus consists of a Sync Center (integrating CDC components like Canal and Debezium) and a Process Center (built on Flink to handle source parsing, SQL processing, and sink routing to databases such as HBase, Elasticsearch, or relational stores).

HA Solution

Both Sync Center and Process Center are natively high‑availability; node failures trigger automatic failover without user impact.

Full‑to‑Incremental Sync

DataBus supports full data migration and incremental sync, automatically converting schema, data types, indexes, and partitions when moving from Oracle to MySQL, PostgreSQL, or other targets.

It also allows field‑level, horizontal sharding, and vertical splitting across heterogeneous stores.

Sharding Support

DataBus can read sharding strategies from middleware such as ShardingSphere and automatically establish horizontal sharding sync links.

3 Technical Highlights

3.1 Flexible, Extensible Sync Architecture

DataBus supports a wide range of source and sink engines, handling massive data volumes, transformation, validation, and in‑process computation.

Three manager nodes coordinate multiple worker nodes that run plugins for frameworks like DataX, Kettle, and Canal.

3.2 High Performance with Heterogeneous Stores

DataBus can ingest, compute, and write to distributed stores such as Oracle, MySQL, PostgreSQL, TiDB, and Elasticsearch, achieving 60‑80k rows per second per 32‑core node.

3.3 Easy‑to‑Use Visual Interface

Users define source and target endpoints via a drag‑and‑drop UI; DataBus automatically creates the sync pipeline, monitors performance, and allows fine‑grained field selection and real‑time SQL processing.

4 Application Scenarios

Distributed Transaction Refactoring

DataBus captures large Oracle transactions, splits them across multiple databases, and preserves transactional consistency at the application layer.

Optimized Query Services Across Multiple Stores

By linking front‑end databases with back‑end data‑platforms (ES, HBase, ClickHouse, StarRocks), DataBus provides second‑level sync for high‑performance cross‑store queries.

Monolith to Micro‑service Migration

During gradual migration, DataBus synchronizes data between legacy monolith databases and new micro‑service databases, ensuring seamless operation.

Data‑Platform Construction

DataBus enables real‑time data‑platform services, allowing front‑end databases to feed heterogeneous storage engines for analytics and reporting.

Cross‑Source Join and Computation

DataBus performs real‑time joins and calculations between sources before writing to the final sink, reducing downstream processing load.

Case: Lufax

Lufax uses DataBus to synchronize MySQL with StarRocks, achieving automatic, second‑level data propagation for new tables without manual intervention.

Case: Ping An Pension

Ping An Pension leverages DataBus to migrate massive Oracle workloads to micro‑service architectures, maintaining data consistency and supporting incremental sync during the transition.