Design and Implementation of a Lakehouse‑Integrated Data Platform for Financial Innovation by Shuxin Network
This article presents Shuxin Network's practical experience in building a cloud‑native, lakehouse‑integrated data platform for the financial sector, covering architecture evolution, challenges of domestic‑innovation (信创), the DataCyber solution, core components, deployment roadmap, and real‑world case studies.
The article shares Shuxin Network's practice in the financial innovation (信创) domain, focusing on the architecture of a lakehouse‑integrated data platform.
Architecture Evolution : It outlines four stages of big‑data infrastructure—data warehouse, data platform (Hadoop), data middle‑platform, and cloud data platform—highlighting trends such as cloud‑native integration, lake‑warehouse convergence, and storage‑compute separation.
Financial Innovation Challenges : The piece discusses the need for domestic‑technology (信创) adaptation, high licensing costs of foreign solutions, and the complexity of component compatibility, emphasizing security, performance, stability, and hybrid deployment requirements.
DataCyber Platform : Introduces the cloud data intelligence platform DataCyber, designed to be open‑source, cloud‑native, and multi‑tenant, with four main products—CyberEngine (data‑engine foundation), CyberData (one‑stop data development), CyberAI (machine‑learning), and CyberMarket (cross‑tenant data exchange).
Core Technical Components : Details include the CyberLakehouse engine (storage, compute, management layers), support for engines like Hive, Spark, Flink, StarRocks, Presto, and integration of data‑lake formats (Hudi, Paimon). It also covers unified metadata (CyberMeta), scheduling (CyberScheduler), integration (CyberIntegration), and market (CyberMarket) components, all adapted for domestic hardware and operating systems.
Deployment Path : Provides a six‑step migration roadmap—building a unified management platform, pilot selection, resource planning, data migration and verification, stress testing, and gradual cut‑over.
Case Studies : Describes two financial implementations: a bank migrating from Cloudera CDH to a domestically‑adapted lakehouse architecture, and a provincial financial service platform integrating multi‑agency data with secure sandboxes and AI capabilities.
The article concludes with thanks to the audience.
DataFunTalk
Dedicated to sharing and discussing big data and AI technology applications, aiming to empower a million data scientists. Regularly hosts live tech talks and curates articles on big data, recommendation/search algorithms, advertising algorithms, NLP, intelligent risk control, autonomous driving, and machine learning/deep learning.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.