Real-Time Data Architecture Evolution for a Complex Supply Chain

The article describes the evolution of the real‑time data architecture used by Dewu's supply‑chain platform, highlighting the challenges of massive, multi‑dimensional data and the need for low‑latency analytics.

Background : The supply‑chain involves JIT inventory, consignment, brand business, and reverse logistics, generating huge volumes of data across many stages. Traditional MySQL reporting became slow due to large table joins, index bloat, and difficulty supporting real‑time analytics.

Early Solutions :

Using AnalyticDB for MySQL (Adb) to sync tables via DTS and leverage its MySQL‑compatible join capabilities. Works for simple joins but struggles with complex queries and high resource consumption.

Building a wide table with Canal + Otter to capture CDC and write aggregated data back to MySQL. Improves single‑table query speed but introduces coupling, debugging complexity, and limited scalability.

Real‑Time Architecture 1.0 : Adopted Flink + Kafka + ClickHouse. ClickHouse provides high‑throughput inserts and fast scans for large, simple models, but lacks upsert support, has weak update/delete semantics, and mixes compute with storage, causing write‑side contention.

Real‑Time Architecture 2.0 : Switched to Flink + Kafka + Hologres, which offers upsert, compute‑storage separation, better join capabilities, partitioning, and backup mechanisms. This setup now serves thousands of daily reports and handles billions of rows of export data.

Key Operational Issues :

Choosing an appropriate segment_key (ordered time field) to limit file scans and improve query performance.

Batch‑stream fusion: offline deduplication by key, merging with streaming data using last_value , and using union all + group by as a join alternative.

Join operator disorder: hash‑based sharding caused out‑of‑order output when header IDs changed. The fix is to join header → detail first, obtain a stable detail_id , then join back to guarantee order.

Code example for the join fix:

insert into sink
Select detail.id,detail.header_id,header.id
from detail
left join (
    Select detail.id AS detail_id,detail.header_id,header.id
    from header 
    inner join detail
    on detail.header_id  =  header.id 
) headerNew
on detail.id  =  headerNew.detail_id

Productivity Tools :

Flink code generator inspired by MyBatis Generator, using templates to produce Flink SQL automatically.

Visual platform that allows online SQL editing, one‑click deployment, caching, and lock‑queue mechanisms for high‑traffic scenarios.

Future Directions include separating compute (e.g., using a memory‑compute engine) from storage, leveraging Apache Hudi for lake‑house integration, and further reducing the “impossible triangle” of latency, cost, and flexibility.