Xiaohongshu transformed its data platform from a simple ClickHouse‑based ad‑hoc analysis to a Lambda‑style architecture and finally to a lakehouse with generic incremental compute, cutting architecture complexity, resource and development costs by one‑third while delivering second‑level queries over trillions of rows.
The article details Xiaohongshu's evolution from a simple ClickHouse‑based analytics layer to a Lambda‑enabled 2.0 stack and finally a Lakehouse‑based 3.0 architecture, showing how each iteration reduced infrastructure complexity, resource consumption and development effort by roughly one‑third while supporting trillions of daily events and AI‑driven use cases.
The article details Xiaohongshu's four‑stage data‑platform evolution—from a simple ClickHouse ad‑hoc setup to a Lambda‑based 2.0 design and finally a lakehouse‑driven 3.0 architecture—highlighting the adoption of general incremental compute, cost‑reduction to one‑third, performance gains of up to ten‑fold, and the SPOT standards that guide the new system.
Xiaohongshu transformed its data platform from a simple ClickHouse‑based analytics stack to a unified lakehouse with generic incremental compute, cutting architecture complexity, resource cost, and development effort by roughly one‑third while supporting petabyte‑scale, sub‑second queries across its 350 million‑user app.
This article details Xiaohongshu's journey from a ClickHouse‑based batch analytics stack to a unified lakehouse architecture powered by generic incremental computing, showing how the company reduced architecture complexity, resource consumption and development effort each to roughly one‑third while supporting trillions of daily events with sub‑10‑second query latency.
This article explains the six key types of architecture—business, product, application, data, technical, and project—detailing their core components, purposes, and examples, and shows how they interrelate to form a complete system design framework for organizations.
This article analyzes JD's end‑to‑end upgrade of its retail coupon search infrastructure, detailing the business drivers, data‑skew challenges, the shift from dual KV and batch pipelines to a unified stream‑batch model built on Apache Doris, and the resulting performance, resource and stability gains across multiple scenarios.
This article details Xiaohongshu's data platform evolution from a simple ClickHouse‑based ad‑hoc system to a Lambda‑style architecture and finally a lakehouse solution, highlighting how the adoption of a new incremental computing model reduced architectural complexity, resource consumption and development effort each to roughly one‑third while delivering sub‑second query performance on petabyte‑scale data.
This article analyzes Xiaohongshu's data platform evolution—from a simple ClickHouse‑based analytics layer to a Lambda architecture and finally a lakehouse design—highlighting how adopting a new incremental computing model reduced architecture complexity, resource consumption, and development effort each to roughly one‑third while delivering sub‑second query performance on petabyte‑scale data.
A McKinsey report reveals that while two‑thirds of enterprises have tried agentic AI, less than 10% achieve scalable value, and robust, modern data architectures—built on seven concrete principles and a four‑step implementation plan—are the decisive factor.
The article dissects the four‑layer data architecture that transforms raw maritime charts into structured knowledge, enables large models to reason about navigation scenarios, and combines algorithmic route screening with LLM‑based refinement to deliver safe, compliant, and efficient autonomous ship routing.
The article examines costly data platform failures—such as a $40 million payroll system collapse and a healthcare.gov outage—highlighting why traditional data middle platforms become data swamps, then explains how Palantir’s ontology approach, with its three‑layer semantic, dynamics, and decision architecture, can turn data into actionable insights and achieve triple‑digit ROI.
The article analyzes costly data‑platform failures—such as a $40 million school‑district payroll system and a collapsed Healthcare.gov launch—identifies the root cause as ineffective data middle platforms, and explains how Palantir’s ontology‑based three‑layer architecture (semantic, dynamics, decision) transforms raw data into automated business actions, delivering measurable ROI across multiple industries.
This case study details how Vivo’s marketing automation platform evolved its data‑driven architecture—from a Presto‑based wide‑table design, through a Bitmap optimization, to a StarRocks migration—addressing performance bottlenecks, reducing resource costs, and enhancing data security.
The article examines why costly traditional data middle platforms often become data swamps, contrasts them with Palantir's ontology‑based approach that acts like a navigation system, and outlines a three‑layer architecture that turns data into automated business actions, delivering multi‑hundred‑percent ROI.
Fanatics unified its fragmented data stack by building a StarRocks‑powered Lakehouse on Apache Iceberg, replacing Redshift, Snowflake, Athena, and Druid, which cut costs by up to 95%, delivered sub‑second dashboard queries on petabyte‑scale data, and enabled real‑time and historical analytics on a single platform.
This article explains how Taotian Group unified real‑time and offline data using Paimon as lake storage and StarRocks for high‑performance OLAP, eliminating costly sync pipelines, cutting refresh time by about 80%, saving nearly ten million yuan annually, and detailing the architecture, cluster safeguards, configuration tweaks, monitoring, and future roadmap for large‑scale promotional events.
This article details iQIYI's evolution from a tightly coupled private‑cloud Kafka setup to a cloud‑native AutoMQ architecture, describing the challenges of scaling, the development of the Stream platform and Stream‑SDK, the migration to hybrid and public‑cloud Kafka, and the resulting cost and elasticity improvements.
This article explains the definitions, core components, purposes, and real‑world examples of six architecture layers—business, product, application, data, technical, and project—and shows how they interrelate to form a complete system design framework.
The article explores the challenges faced by traditional DBAs, redefines the DBA role as a blend of data, business, administration and architecture, presents multiple real‑world cases, and shows how AI and a DBA² mindset can dramatically improve efficiency and career prospects.
This article explains how Xiaohongshu, a lifestyle community with over 350 million monthly users, transformed its data platform from a traditional Lambda architecture to a next‑generation incremental computing model, reducing architectural complexity, resource consumption and development effort each by roughly two‑thirds while supporting massive real‑time and offline data demands.
Facing PB‑scale user behavior data and millions of feature dimensions, the platform transformed its search, advertising, and recommendation pipelines by adopting a distributed, configurable‑service architecture that delivers high‑throughput streaming, elastic storage, rapid feature iteration, and robust fault‑tolerance for AI‑driven personalization.
This article shares Ant Group’s practical experience in large-scale data governance, outlining four main topics—overall governance overview, data quality management, data storage-processing governance, and future considerations—while emphasizing the five critical aspects of architecture, security, compliance, quality, and value that drive effective big-data operations.
This article shares a data‑governance roadmap for small and medium banks, covering industry pain points, high‑quality data sets, a three‑step governance path, data standards, metadata management, master‑data strategy, business data modeling, a hybrid Greenplum‑Hadoop platform, quality monitoring, and a maturity assessment framework.
Ant Group shares its practical experience in big data governance, outlining five key focus areas—architecture, security, compliance, quality, and value—through four structured sections and detailed discussions on data quality and storage governance, while also exploring future challenges and the economics of data.
This article traces the three‑generation evolution of data architecture—from the structured‑data era of data warehouses, through the flexible, multi‑format data lake, to the unified lakehouse model—explaining the drivers, benefits, challenges, and future trends shaping modern data platforms.
ICBC’s Software Development Center built an AI‑powered, multi‑layer data platform and decision engine that enables real‑time, precise risk monitoring and automated response for credit‑card operations, dramatically improving detection speed, coverage, and warning quality while supporting a full‑process intelligent risk‑control loop.
This article explains why a robust tag system is essential for recommendation and mining strategies, outlines the hierarchy of entity, concept, and theme tags, and provides practical principles, architecture, and step‑by‑step methods for constructing and managing tags in large‑scale data platforms.
This article explains how ElasticSearch serves as a core middleware for large‑scale architectures, covering its role in search engines, log analysis with the ELK stack, massive data querying, and even as an independent database system, illustrated with practical examples and diagrams.
Lazada transformed its e‑commerce product selection by building a unified, real‑time platform on Alibaba Cloud Flink and Hologres, overcoming data silos, freshness delays, and high‑throughput challenges to enable millisecond‑level decisions across six Southeast Asian markets.
JD Retail faces growing data volume, redundant models, and resource‑intensive storage, prompting a comprehensive data‑governance strategy that defines standards, streamlines architecture, isolates development, and optimizes compute and storage costs, ultimately enabling more efficient, secure, and agile data operations across the enterprise.
JD Retail confronts growing data volume, redundant models, shared account risks, and rising storage costs, and responds with a comprehensive data governance framework that standardizes data, streamlines architecture, isolates development, and optimizes resources to achieve efficient, secure, and cost‑effective data operations.
Facing daily data growth from millions to 150 billion records, Joydata‑U transformed its analytics platform through three architectural stages—Hadoop, Hadoop + Trino, and finally StarRocks—introducing resource isolation, Flat JSON acceleration, and Bitmap indexing to cut query latency by up to seven times and achieve sub‑2‑minute data freshness across BI, ad‑tech, game analytics, and CRM workloads.
This article details Beike's data‑fusion architecture evolution, covering industry trends, multi‑stage Hadoop upgrades, storage cost optimization with erasure coding, remote shuffle integration, GPU‑centric training stability, and future hybrid‑cloud strategies, while also sharing organizational and operational lessons learned.
This guide presents a series of expert case studies and insights on how Cloud Lakehouse solutions enable real‑time, fully managed multidimensional data analysis, improve user data experiences, balance performance and cost, and power large‑scale IoT and big‑data platforms across industries.
This article shares a collection of PPT slides that showcase classic data architecture and data model design methods, offering valuable insights for building data middle platforms and master‑data‑driven governance solutions.
This article compiles a series of expert case studies and insights on real‑time intelligent fully‑managed Lakehouse technology, illustrating how companies such as SalesEasy, Chang’an Auto, Kuaishou, Tencent, and JD.com leverage lakehouse architectures to achieve advanced multi‑dimensional analytics, cost‑performance balance, and effective data governance in the digital economy.
AI-era data architecture must handle massive, multimodal datasets with real-time processing, prioritize data quality over quantity, support scalability, provenance, and native ML/AI integration, while addressing governance, security, and ethical challenges through emerging technologies like data fabric, mesh, and federated learning.
This article explains the data resourceization stage, outlines seven core data management functions, introduces the DataOps paradigm to overcome traditional inefficiencies, and showcases real‑world case studies and a panoramic data architecture view for modern enterprises.
This article presents a comprehensive visual guide to enterprise digital transformation, covering overall digital planning, application system architecture, data architecture, information security architecture, and digital organization and governance, illustrating each layer with detailed diagrams to aid strategic implementation.
This article shares Ant Group's practical experience and evolution of data architecture, covering theoretical foundations, analysis of current challenges, proposed upgrade solutions based on domain‑driven design, and future outlook through complex network theory to improve scalability, governance, and resilience of large‑scale data systems.
This article shares the foreign bank's big data implementation journey, covering background and goals, overall planning and architecture, practical insights, phased rollout, data governance, security, and Q&A, illustrating how a unified data platform, storage‑compute separation, and AI‑driven tools drive business innovation.
The article details the background, architectural analysis, multi‑tenant redesign, engine selection enhancements, compatibility adaptations, stability fixes, containerized deployment, performance optimizations, and measurable business outcomes of the Intelligent Acceleration Engine upgrade using Apache Kyuubi and StarRocks within the 58 big data platform.
This article surveys the evolution of modern data technologies—from the 4V characteristics of big data and the limitations of traditional relational databases, through the rise of NoSQL and polyglot persistence, to embedding‑driven vector search, hybrid retrieval and RAG, illustrating how each paradigm frees applications from data constraints.
DataFunSummit2024, co-hosted by Bilibili, convenes industry experts, scholars, and enterprise leaders across six forums to discuss next‑generation data architecture, showcasing Bilibili’s Iceberg‑based stream‑batch innovations, AI‑BI analytics, NoETL practices, and emerging alternatives to Lambda architecture.
This article reviews the emerging lakehouse architecture, outlines its core frameworks such as Hudi, Iceberg, Paimon, Flink, and Doris, discusses their storage‑compute separation, read‑write optimizations, and highlights how companies of different sizes adopt these technologies based on cost, efficiency, and specific business scenarios.
Amid growing data demands, this article explains the data lake technology maturity curve, detailing lake‑warehouse architectural patterns, design principles, core functionalities, and the four leading open‑source solutions (Hudi, Iceberg, Delta Lake, Paimon) to guide enterprises in building flexible, scalable, and governed data platforms.
The article chronicles the author’s 30‑year programming career, detailing early experiences, the evolution from JavaScript projects to large‑scale big‑data architectures, cloud migration, business‑agnostic framework design, interactive analytics, and reflections on becoming an independent software architect.
This article reviews the current state of data platforms, analyzes the third wave of data infrastructure evolution driven by databases, big data and generative AI, proposes next‑generation lakehouse and cloud‑native architectural directions, and outlines future trends and unresolved challenges for AI‑centric data platforms.
This article explains the concept, design principles, and core components of a data middle platform, detailing its overall, functional, layered, logical, and data architectures, as well as the specific platforms for data collection, processing, organization, governance, quality, sharing, and visualization, illustrated with diagrams.
Data architecture governance, encompassing standards, security, modeling, quality, and lifecycle management, is essential for digital transformation in fast‑growing industries like express delivery, and this article outlines current challenges, traditional approaches, and a practical, phased methodology with platform support to implement effective governance.
This article explores the DIKW hierarchy, explains how data evolves into information, knowledge, and wisdom, examines traditional data models and products, critiques existing multi‑system architectures, and proposes a new distributed Data Warebase that unifies structured, semi‑structured, and vectorized knowledge to enable intelligent data-driven applications.
The forum presents a series of technical talks on multi‑cloud data architecture, covering Xiaomi’s lake‑warehouse practice, cross‑border e‑commerce data platforms, Alluxio‑based machine‑learning acceleration, Qichacha’s cost‑effective data solutions, and Kuaishou’s Flink on Kubernetes migration, highlighting strategies, implementations, and audience benefits.
The article presents a comprehensive case study of XinXuan Group’s data ecosystem, detailing its business background, data platform construction, governance framework, technical architecture, and practical analytics applications that support live‑commerce operations and strategic decision‑making.
This article explains the TOGAF framework and its 4A architecture model—Business, Technology, Application, and Data Architecture—detailing each domain's definition, purpose, value, and step‑by‑step guidance for creating enterprise architecture diagrams to align strategy, processes, and technology.
This article presents a detailed case study of how EasyPoint built a scalable, stable advertising platform using ClickHouse Enterprise, covering company background, data architecture with Kafka and Druid, ClickHouse advantages, serverless resource scaling, and extensive performance benchmarks.
This article explains the concept of query separation, outlines its suitable scenarios, compares implementation methods such as synchronous, asynchronous, and binlog approaches, discusses storage system choices like MongoDB, HBase, and Elasticsearch, and addresses consistency and operational challenges when decoupling read workloads from write workloads.
This article describes how 58.com designed and implemented a large‑scale user profiling platform using ClickHouse, covering system overview, core modules, major challenges of scale, complexity and performance, and the detailed storage, query, and optimization techniques applied to meet business needs.
This article provides a detailed overview of vector database structures, their evolution, enterprise challenges, functional features, future trends, and key use cases, illustrating how they serve as the memory engine for large AI models and support multimodal data processing in modern data architectures.
This article shares Ant Group's extensive experience in big data governance, detailing the overall data governance framework, data quality management, data storage governance, and future considerations, illustrated with practical cases and strategies for ensuring compliance, reliability, and cost efficiency.
This article outlines Didi’s best‑practice approach to constructing a global ride‑hailing data metric system, covering business scenarios, metric categories, pain points such as definition and technical challenges, and a comprehensive solution involving organizational structure, processes, model design, tooling, timezone handling, and governance.
DingoDB is a multi‑modal vector database that unifies storage and analysis of structured, semi‑structured and unstructured data through a Raft‑based distributed architecture, offering MySQL‑compatible SQL, high‑performance APIs, automatic sharding, real‑time index optimization, and hybrid scalar‑vector queries for enterprise knowledge bases, LLM memory, and real‑time decision‑making.
This presentation by senior data warehouse engineer Huang Luo details Yipay's end‑to‑end data warehouse build, covering background challenges, governance framework, platform development, layered architecture, naming standards, monitoring, and future plans, offering practical insights for data engineers, architects, and business stakeholders.
Vivo’s trillion‑scale data platform evolved into a dual‑active real‑time and offline architecture that leverages multi‑datacenter clusters, Kafka/Pulsar caching, a unified sorting layer, HBase‑backed dimension tables, and micro‑batch Spark jobs to deliver low‑cost, high‑performance processing, 99.9% availability, and 99.9995% data‑integrity.
This article outlines the evolution of software architecture styles, describing various architectural domains and sub‑domains—from web and mobile applications to integration, data, and analytics architectures—and their typical implementations, illustrated with a detailed classification table.
This article details how CaoCao Mobility transformed its ride‑hailing platform by replacing a traditional Lambda architecture with an enterprise‑grade real‑time data warehouse built on Hologres and Flink, covering business motivations, architectural design, component capabilities, performance optimizations, operational safeguards, and future roadmap.
Designing a data governance system involves defining data architecture, standards, and master data, documenting mechanisms, aligning them with actual business and analytical data processes, forming joint teams for specialized governance activities, and institutionalizing these practices through formal processes to sustain continuous data capability across the enterprise.
This article explains what a data lake is, its origins, key characteristics such as collecting all data, enabling diverse user access, and flexible processing, compares it with traditional data warehouses, discusses cost advantages, potential pitfalls like data swamps, and outlines best‑practice considerations for enterprise adoption.
The article explains why unified data architecture is essential to avoid duplication and inefficiency, discusses differing performance trade‑offs among batch, streaming, and interactive analytics, introduces an incremental computation model that unifies these modes, and invites readers to a Dec 19, 2023 technical sharing event.
Data lakes and data warehouses serve different purposes in big‑data architectures; this article explains their definitions, core attributes, five major distinctions—including data retention, type support, user coverage, adaptability, and insight speed—and offers guidance on selecting or combining the two approaches.
This article explains how Kuaishou's e‑commerce team built a unified OLAP‑based data platform—covering data ingestion, consistent dimensional and fact layers, metric management, and real‑time services—to address rapid growth, metric inconsistency, and operational inefficiencies across multiple business scenarios.
This article explains why Kuaishou built a data lake, outlines the shortcomings of its previous Lambda architecture, describes the adoption of Apache Hudi for unified batch‑stream processing, and details the five major technical challenges and the corresponding solutions implemented to improve performance, consistency, and operational reliability.
This article explains the data mesh architecture—its domain‑driven, decentralized design, key concepts, benefits over traditional data lakes, a scoring method to evaluate suitability, and the importance of observability for trustworthy data products.
This article provides a comprehensive overview of GPFS (IBM Spectrum Scale) and Ceph, detailing their architectures, core components, deployment models, use cases, performance considerations, and a comparative analysis with GlusterFS, while also discussing future trends in distributed storage.
Manbang Group shares its practical experiences and insights on real-time computing, multi‑cloud platform architecture, data warehousing with Flink and Holo, real‑time decision and feature platforms, and future plans for scaling these systems to support logistics and recommendation algorithms.
This article traces the evolution of data architecture from early BI systems through traditional big‑data stacks, streaming, Lambda and Kappa designs, and explains how a unified stream‑batch model simplifies development while keeping logic consistent across data‑analysis and pipeline applications.
This article examines the middle‑platform concept in large companies, especially data departments, outlining its benefits such as resource centralisation, reduced duplication and experience sharing, while also highlighting drawbacks like slow response, excessive authority, and the need to bring platform capabilities closer to business units.
The panel discusses the three‑stage evolution of data platforms, compares US and Chinese market dynamics, evaluates Snowflake’s success factors, and outlines the criteria and opportunities for a China‑specific Snowflake‑like solution, while also sharing investment perspectives on data‑driven startups.
This article explores the evolving concept of data lakes, compares them with traditional data warehouses across storage, modeling, tooling, and user roles, and examines the emerging lake‑warehouse integration, highlighting why both remain essential in modern big‑data architectures.
The article explains the concept, benefits, and detailed modular architecture of a data middle platform, covering data storage, acquisition, processing, governance, security, and operation frameworks, and illustrates how enterprises can build and evolve such platforms to turn data into valuable services.
This article explains data architecture and data modeling fundamentals, presents DAMA DMBOK concepts, outlines four practical solutions for model design, standard management, automated change control, and business mapping, and shares an enterprise manufacturing case study with Q&A on governance and efficiency.
This article explains why channel data products are needed, outlines the stages of growth business, describes how to design and implement a channel data product—including its architecture, database schema, and operational workflow—and concludes with a practical summary and Q&A.
This technical presentation reviews the historical development of big data platforms, outlines the four generations of cloud data platform architectures, details the modern cloud‑native stack—including unified metadata, scheduling, and integration systems—and showcases a real‑world industrial manufacturing case with a Q&A session.
The article explains how enterprises can build a comprehensive big data analytics platform—covering data collection, storage, computation, and decision layers—by clarifying business scenarios, choosing appropriate on‑premise or cloud deployment, selecting suitable architectures such as Lambda/Kappa, and addressing component choices and emerging technical trends.
This article provides an in‑depth overview of enterprise data architecture, covering data architecture components, business drivers, modeling techniques, measurement metrics, key concepts, tools, and best practices for designing, implementing, and maintaining robust data models and databases.
This article details the evolution of Tencent Music's content data platform from version 1.0 to 4.0, describing the migration from ClickHouse to Apache Doris, the introduction of a semantic layer, optimization of data ingestion, query performance, and cost reduction strategies that dramatically improved data timeliness, operational efficiency, and storage costs.
This article explains data governance by presenting authoritative definitions, outlining a five‑level requirement hierarchy inspired by Maslow, and detailing practical considerations for stability, security, usability, quality, and cost‑value to help organizations implement effective data governance frameworks.
This article details how ZuanZuan leveraged ClickHouse as the core OLAP engine for its massive self‑service analytics platform, covering OLAP engine selection criteria, system architecture, real‑world use cases, performance tuning, operational challenges, and future development plans.
iQIYI’s data lake combines public‑cloud and private storage with Apache Iceberg’s snapshot‑based table format to enable near‑real‑time, unified batch‑and‑stream analytics, reducing costs, simplifying architecture, and improving data freshness across use cases such as log collection, audit, pingback, and member order processing.
This article details how Tongcheng Data Science built a real‑time analytical data warehouse using Apache Doris, covering business scenarios, the evolution from a legacy 1.0 architecture to a Doris‑based 2.0 design, deployment topology, development workflow, operational benefits, and future roadmap.
Facing massive daily data volumes and complex, ad‑hoc analytical needs, Zhaozhuan’s engineering team evaluated multiple OLAP engines and chose ClickHouse, then built a four‑layer self‑service analytics platform, detailing architecture, use‑cases, performance tuning, large‑scale joins, and future roadmap challenges.
This article examines why ClickHouse was chosen as the OLAP engine for a massive self‑service analytics platform, describes the system architecture, shares concrete memory and performance tuning parameters, and outlines current challenges and future roadmap for large‑scale real‑time data analysis.
This article examines the selection of ClickHouse as the OLAP engine for a self‑service analytics platform, describes the platform’s architecture, details memory and performance tuning techniques, discusses large‑scale join handling, and outlines current challenges and future development directions for ClickHouse.
The article explains how to formally name data schemas, structures, and models, clarifies the differences between internal, external, logical, and physical schemas, and proposes a systematic hierarchy—including strategic and tactical schemas—to improve data architecture and reduce confusion in enterprise data design.
This article explains what data architecture is, why it is essential for linking business and technology, outlines its main components such as data models, data flows, value streams and standards, and discusses emerging trends toward service‑oriented, consumption‑focused data architectures.
This article details Baicaowei’s journey from a Hadoop‑based data platform to a modern StarRocks‑driven architecture, illustrating how digitalization, evolving business needs, and streamlined data pipelines empower the fast‑moving consumer goods sector through efficient data collection, modeling, and analytics.
This article traces the history of data lakes from their 2010 inception with Hadoop through cloud‑native object storage, lakehouse formats like Delta Lake, and Alibaba Cloud's multi‑layer solution, outlining key architectural stages and practical construction challenges for enterprise‑grade implementations.
The TDengine Developer Conference in Beijing unveiled the open‑source, cloud‑native TDengine 3.0, detailing its revolutionary architecture that tackles high‑cardinality challenges, introduces RAFT‑based distribution, and showcases real‑world IoT and IT‑operations case studies where enterprises dramatically improved performance and reduced costs.
This article explores the evolution and architecture of cloud‑native data platforms, covering their historical roots, modern components such as storage layers, ingestion, processing, metadata, and consumption, and offers practical guidance on selecting tools, designing pipelines, and implementing best‑practice strategies for scalable, flexible data infrastructure.
In this interview, Tencent Cloud expert Zhang Zhigang explains the fundamentals and key technologies of lakehouse architecture, discusses how cloud‑native practices enhance its performance and operability, and offers practical advice for big‑data professionals ahead of the 2022 GIAC Global Internet Architecture Conference in Shenzhen.
