This article details the migration from a traditional Hive‑based data warehouse to a lakehouse architecture using Apache Hudi, covering the original Lambda setup, its pain points, lake‑vs‑warehouse differences, Hudi features, integration challenges, practical solutions, and future roadmap.
This article traces a decade of Amazon Redshift’s evolution, detailing its shift from a traditional MPP warehouse to a fully cloud‑native Serverless architecture, exploring its underlying innovations, key features such as Concurrency Scaling, built‑in ML, Data Sharing, and offering practical best‑practice guidance for real‑time analytics across diverse industry scenarios.
The article explains how enterprises can build a comprehensive big data analytics platform—covering data collection, storage, computation, and decision layers—by clarifying business scenarios, choosing appropriate on‑premise or cloud deployment, selecting suitable architectures such as Lambda/Kappa, and addressing component choices and emerging technical trends.
Yidian Tianxia tackled massive daily data volumes and complex analytics by defining a five‑layer data‑warehouse standard, comparing ClickHouse and StarRocks performance, and implementing a unified real‑time/offline architecture with StarRocks, DataPlus, and EasyJob, achieving multi‑fold query speedups and lower operational costs.
This article presents ZhongAn Insurance's experience building real-time multi-dimensional analytics, covering application scenarios, technical challenges, the evolution of their architecture from offline to Flink‑ClickHouse and finally to StarRocks, and the principles guiding their technology selection.
Based on a DTCC2022 presentation, this article explains Apache Doris's high‑performance MPP architecture, its cloud‑native extensions in SelectDB, and how they solve large‑scale log storage and analysis with superior write throughput, storage efficiency, and interactive query speed.
This article explores the evolution of real‑time computing and compares three interactive analytics engines—Impala, Apache Flink, and Slipstream—detailing their architectures, key features, deployment considerations, and why they matter for modern big‑data stream processing.
This article details Kuaishou's one‑stop big data development platform, covering its massive scale, low‑code and real‑time capabilities, multi‑layer architecture, SLA guarantees, diagnostic tools, and future plans to further lower development barriers and democratize data engineering.
The Dolphin engine, built by Alibaba’s Data Engine team, combines Flink and Hologres to deliver ultra‑large‑scale OLAP, streaming, batch, and AI capabilities for real‑time advertising analytics, offering smart materialization, intelligent indexing, and vector recall while supporting millions of advertisers and petabyte‑level data.
This article presents a comprehensive overview of evolving big‑data architectures—from classic offline warehouses to Lambda and Kappa models—and details a streaming‑batch integrated solution that addresses latency, data freshness, and multi‑table join challenges to achieve minute‑level real‑time multi‑dimensional analytics.
This article presents Glacier, OPPO's intelligent data lake solution that builds on Iceberg Table Format to provide real‑time data ingestion, low‑latency queries, advanced indexing, and robust multi‑version management for both structured and unstructured data, tightly integrating with machine‑learning workflows.
This article explains the data and feature engineering challenges of deploying machine learning, introduces OpenMLDB’s open‑source architecture—including offline Spark‑based processing, a high‑availability online engine with dual‑layer memory indexes, snapshot/binlog persistence, and pre‑aggregation techniques—then showcases real‑world case studies and the project’s roadmap.
The article presents a comprehensive risk insight platform built on ClickHouse, Flink, and intelligent algorithms, detailing its architecture, technical challenges, solutions, real-time data modeling, practical applications in fraud detection and user behavior analysis, and future optimization directions.
This article explains the challenges of deploying intelligent features on mobile clients and describes Ant Group's edge computing container, its three‑layer architecture, real‑time compute, feature, and decision engines, and the low‑code platform that enables fast, stable, and scalable AI solutions on devices.
The article explores Apache Flink’s eight‑year journey to becoming a top‑level Apache project, Alibaba’s extensive contributions, the rise of stream‑batch unified computing, its impact on real‑time data integration, cloud‑native deployment, and the emerging Flink‑based data‑warehouse and serverless solutions.
This interview with Alibaba researcher Wang Feng (aka Mo Wen) explores Apache Flink’s eight‑year journey to top‑level Apache status, its unified stream‑batch architecture, the rise of Flink Table Store and CDC, and how cloud‑native deployments are reshaping real‑time big data processing.
This article outlines the background and terminology of data warehousing, presents a formula for evaluating warehouse value, and details the team's half‑year efforts—including architecture selection, quality assurance, stability governance, and data‑value externalization—to improve efficiency, quality, stability, and cost in real‑time data services.
This article outlines the evolution of StarRocks’ storage engine—from its real‑time update capabilities and primary‑key model challenges to recent optimizations like persistent indexes, partial column updates, conditional updates, high‑frequency import improvements, DML support, and future plans for separating primary and sort keys, introducing row‑store, and enhancing materialized view support.
Alibaba Cloud announced that its ODPS platform has been upgraded into an integrated big‑data solution that supports massive batch jobs, real‑time analytics, and AI workloads, delivering record‑breaking performance and enabling use cases from smart city traffic optimization to accelerated autonomous‑driving model training.
The article presents ByteDance's challenges with massive real‑time data processing and describes how they integrated a streaming data warehouse with Flink Table Store, cloud‑native architecture, and real‑time service analysis to achieve low‑latency, high‑throughput analytics and end‑to‑end consistency.
The article presents a comprehensive overview of Tencent Cloud's metering and billing system, detailing the billing models, multi‑dimensional data analysis, real‑time data‑warehouse construction, operator orchestration, hot‑key handling, smooth upgrade strategies, and future evolution directions for large‑scale cloud services.
This article explains how ByteHouse extends ClickHouse by introducing the UniqueMergeTree engine, which adds full upsert and delete support, improves query performance, and enables real‑time audience selection scenarios while preserving scalability and reliability.
This article explains the architecture of Apache Doris, introduces Apache Hudi as a data‑lake format, compares Lambda and Kappa approaches, and details the design, implementation steps, and future roadmap for querying Hudi tables directly from Doris.
StarRocks is a high‑performance MPP database whose simplified FE/BE architecture, fully vectorized engine, and CBO optimizer enable fast multi‑table joins, while its partition‑bucket‑tablet storage model supports real‑time metric services and dashboard migrations, accompanied by practical troubleshooting guidance and upcoming enhancements.
Shopee tackled its data‑system bottlenecks by customizing Apache Hudi to provide unified stream‑batch integration, efficient state‑detail snapshots, and low‑latency wide‑table generation, using CDC‑based bootstrapping, COW/MOR tables, savepoints and partial updates, which cut latency to ten minutes, lowered resource use, and yielded several community‑backed enhancements.
JD Logistics leverages StarRocks to create the Udata unified query engine, addressing data silos, low performance, and high maintenance costs by integrating data services and analytics, enabling low‑code data service generation, high‑speed federated queries, real‑time updates, and future data‑lake and resource isolation capabilities.
This article presents a comprehensive overview of how TiDB's HTAP capabilities enable real‑time multi‑source data processing for financial intelligent risk control, detailing the overall risk‑control architecture, digital transformation challenges, TiDB‑based solutions, practical implementations, and future outlooks.
This article introduces the practice of Change Data Capture (CDC), explaining how capturing only data changes can feed downstream systems and data warehouses in near real‑time, reducing load on the source database, improving reporting latency, and supporting scalable, reliable analytics pipelines.
This article introduces Apache Doris, an MPP analytical database, and explains how it integrates with the Hudi data lake format, covering architectural features, design choices, implementation steps including external table creation and query processing, and outlines future enhancements for supporting MOR snapshots and incremental queries.
This article explains what Elasticsearch is, how it solves database acceleration, log observability, and data analysis problems, details its core components and underlying engine features, compares its strengths and weaknesses, and presents classic application scenarios and a real‑world case study integrating Elasticsearch with Flink for large‑scale log analytics.
Apache Druid is a distributed, column‑store OLAP engine designed for massive real‑time data ingestion and sub‑second queries; this article explains its LSM‑tree‑inspired architecture, DataSource and Segment structures, memory‑based querying, practical deployment steps, common pitfalls, and optimization techniques for high‑throughput analytics.
This article details Qunar's multi‑year journey of designing and evolving a full‑stack BI platform—covering data ingestion, storage, query engines, self‑service analytics, and real‑time OLAP—by iterating through three development phases, selecting technologies such as Impala, Kudu, ClickHouse and Apache Druid, and addressing performance, usability and governance challenges to empower business users with fast, reliable data insights.
This article examines the core trade‑offs of data warehouses—freshness, query latency, and cost—compares offline and real‑time architectures, introduces the concept of a streaming data warehouse, and details how Apache Flink Table Store aims to provide a unified, low‑cost solution.
This article introduces the concept and practice of Change Data Capture (CDC), explaining how it captures database changes to provide real‑time incremental data for analytics and reporting without impacting source performance, and outlines modern CDC methods, challenges, and production‑ready system requirements.
This article introduces the concept of Change Data Capture (CDC), explains why traditional batch reporting strains resources, describes how CDC captures only data changes to keep source databases performant, and outlines modern CDC architectures, production‑ready considerations, and best‑practice guidelines for building reliable data pipelines.
This article explores how combining Flink CDC with StarRocks can streamline real‑time data pipelines, reduce component complexity, support both full and incremental synchronization, and enable efficient OLAP queries and updates for fast, scalable analytics across diverse business scenarios.
This article walks through the four essential stages of building a data project—data collection, modeling, analysis, and application—explaining key principles, common models such as 3NF, star/snowflake, cube, and wide tables, and comparing offline versus real‑time pipelines.
The article explains the fundamentals of Change Data Capture (CDC), describing how capturing DML changes from relational databases like MySQL or PostgreSQL can provide incremental, near‑real‑time data for analytics and reporting while preserving source performance, and outlines modern CDC architectures, transaction‑log based extraction, and production‑ready design considerations.
The article presents a comprehensive overview of Lianyou Technology's car IoT platform, detailing its four‑layer architecture, configurable data ingestion, hybrid cloud storage solutions, real‑time and offline data warehouses, and downstream data applications such as user operation, intelligent recommendation, and data security practices.
This article presents a comprehensive technical walkthrough of Sensor Data's multi‑dimensional analysis platform, covering product architecture, an Impala‑based real‑time query engine, query performance tuning, resource‑estimation strategies, and future plans, with concrete diagrams, test results, and community contributions.
This article explains how StarRocks manages petabyte‑level user behavior logs, ads and orders through a shared‑nothing architecture, tablet‑based data distribution, MPP compute, high‑availability metadata, real‑time mini‑batch ingestion, and online schema changes, enabling 24/7 analytical services for diverse internet companies.
In 2022 the modern data stack will be driven by the rise of analytics engineers, intensified competition between lakehouse and warehouse solutions, growing demand for real‑time analytics, the explosive growth of cloud marketplaces, and the emergence of unified data‑quality terminology, all reshaping data infrastructure and operational practices.
This article describes how eBay applies graph neural networks to real‑time payment fraud detection, covering the anti‑fraud scenario, limitations of traditional GBDT pipelines, challenges of constructing and serving dynamic heterogeneous graphs, the end‑to‑end solution with directed slice graphs and a Lambda‑style architecture, and experimental results comparing GNN with LightGBM.
This article explains TiDB's HTAP architecture, covering industry challenges, the row‑store TiKV and column‑store TiFlash design, MPP integration in TiDB 5.0, and a range of real‑time use cases such as dashboards, reporting, and data‑warehouse pipelines.
This article details ByteHouse’s evolution from ClickHouse, presenting two real‑time analytics use cases, the technical selection process, performance bottlenecks such as write throughput and Kafka consumption, and the engineered solutions—including asynchronous indexing, multi‑threaded Kafka engines, and enhanced Buffer engines—that enable reliable, high‑throughput data processing at massive scale.
Shopee’s engineering team scaled a 100‑node Apache Druid cluster for real‑time OLAP by redesigning the Coordinator load‑balancing algorithm, adding incremental metadata pulls, introducing a segment‑merged result cache, and building exact‑count and flexible sliding‑window operators, while planning cloud‑native deployment.
This interview explores Apache Flink’s evolution toward a Streaming Warehouse, detailing its stream‑batch integration, new CDC‑based data integration, the Dynamic Table storage architecture, and how these innovations aim to simplify and accelerate real‑time big‑data analytics.
ByteDance’s data platform, built over seven years, now handles exabyte-scale data and over 100 million TPS, using a hybrid “middle‑platform + Business Partner” model, custom engines like ClickHouse/ByteHouse, agile governance, and a suite of products to support internal and external businesses, illustrating large-scale big-data engineering practices.
This article details the evolution, architecture, and technical choices of Qunar's BI platform—from early one‑stop reporting to a modular, self‑service system supporting real‑time analytics, multi‑metric calculations, and unified data governance—highlighting challenges, solutions, and performance benchmarks across big‑data technologies.
Facing Hadoop’s minute‑to‑hour query latency on petabyte‑scale data, WeChat partnered with Tencent Cloud to build a ClickHouse‑based real‑time warehouse, adding custom ingestion, query‑optimisation and management tools that deliver billion‑row throughput, sub‑5‑second queries and over ten‑fold performance gains across millions of daily queries.
This article describes AutoHome's real‑time data analysis architecture, the challenges of existing OLAP solutions, the reasons for choosing StarRocks, detailed performance comparisons with Kylin, ClickHouse, Doris, Presto and Spark, and the practical integration of StarRocks with Flink, broker‑load scripts, and monitoring tools.
This article presents Tencent's game big‑data analysis platform, detailing its background, the architecture of the iData engine—including offline multi‑dimensional analysis (TGMars), online portrait analysis (TGFace), and real‑time multi‑dimensional analysis (TGDruid)—application scenarios, performance insights, and future ecosystem and open‑source plans.
This article details Youzan's adoption of ClickHouse for real-time analytics, covering its evolution from Presto, Druid, and Kylin, the system's architecture, deployment strategies, use cases, performance characteristics, limitations, and future roadmap, including integration with Apache Doris and emerging big‑data trends.
This article explains how JD.com designs and evolves its OLAP platform, covering data sources, ingestion, storage, real‑time and offline processing, key challenges such as timeliness, high throughput, consistency, and the solutions implemented to support massive e‑commerce analytics.
This article examines the strengths and weaknesses of Apache Iceberg, explains why Tencent selected it over alternatives, details Tencent’s own enhancements and integration with Flink, Spark, and other engines, and shares multiple real‑world implementations for building enterprise‑grade real‑time data lakes.
JD.com’s customer service team leverages the open‑source MPP database Doris to power real‑time and offline OLAP dashboards, detailing data ingestion pipelines, full‑link monitoring, dual‑stream high‑availability design, dynamic partition management, multi‑layer caching strategies, and performance optimizations applied during the 2020 11.11 shopping festival.
This article shares practical insights on designing and operating a real‑time clickstream data warehouse using Flink for streaming processing and ClickHouse for near‑real‑time OLAP, covering dimensional modeling, layered architecture, Flink‑ClickHouse sink implementation, and data rebalancing strategies.
This guide shows how to build a real‑time UV and PV analytics pipeline on Tencent Cloud Oceanus by connecting a self‑hosted Kafka cluster to Flink SQL, using Redis for deduplicated visitor counts, page view logs, and conversion‑rate calculations via hop windows.
This article shares NetEase’s experience building a user‑profile platform with ClickHouse, detailing the business background, challenges of massive data and complex queries, core table designs, data ingestion, bitmap techniques, performance gains, and future plans for scaling and optimization.
This article details NetEase Games’ evolution of its Flink SQL platform, from the early StreamflySQL v1 template‑JAR approach to the v2 SQL‑Gateway architecture, discussing design decisions, challenges such as metadata persistence, multi‑tenant security, horizontal scaling, and job state management.
ByteDance’s ByteHouse, an enterprise‑grade ClickHouse, powers real‑time recommendation and ad‑delivery analytics at massive scale, detailing two case studies, technical selections, architectural designs, and performance optimizations such as asynchronous indexing, multi‑threaded Kafka consumption, and enhanced buffer engines to ensure data integrity.
The article details how Kuayue Group's big‑data center replaced Presto and ClickHouse with DorisDB, achieving sub‑second query latency, simplifying architecture, and enabling both online and real‑time OLAP analytics across millions of daily requests.
This article examines JD's shift from a traditional Lambda‑based data warehouse to a Delta Lake‑powered real‑time data lake, detailing the challenges of legacy architectures, the evaluation of open‑source table formats, Delta Lake's core mechanisms, and the resulting simplified batch‑stream development workflow.
The article describes how Beike's data platform consolidated multiple OLAP engines into a single DorisDB‑based solution, achieving higher query performance, lower operational complexity, and robust support for diverse analytics workloads such as metrics, dashboards, real‑time updates, and multi‑table joins.
Kudu is an open‑source columnar storage engine from Cloudera that combines high‑throughput batch processing with low‑latency random reads, offering features such as C++/Java APIs, Raft‑based replication, flexible consistency, partitioning, and integration with Hadoop, Spark, Impala, and other ecosystem components.
This article describes how Shuhai Supply Chain upgraded its data warehouse from a complex, high‑cost 1.0 architecture to a streamlined, real‑time solution built around Apache Doris, detailing the motivations, design choices, zero‑code ingestion, metadata management, Flink connector, and the resulting performance gains.
This article explains Apache Kudu’s architecture, schema design, update mechanism, and how NetEase leverages it for real‑time data ingestion, dimension table joins, data‑warehouse ETL, and AB‑testing, while also discussing encountered issues and upcoming feature requests.
This article explains how Beike adopted the Druid OLAP engine to handle massive real‑time and offline query workloads, detailing its four‑component architecture, key technologies such as deep storage and metadata storage, practical optimizations for data ingestion, query caching, dynamic throttling, timeout control, and a roadmap for future enhancements.
This article explains how to construct a lake‑house architecture using Apache Iceberg, detailing the migration from Hive, Flink‑SQL integration, proxy user support, CDC handling, copy‑on‑write sinks, and the resulting benefits for near‑real‑time data visibility and unified batch‑stream processing.
This comprehensive guide explains the concepts, architecture, and best practices of data middle platforms, covering data aggregation, ingestion tools, offline and real‑time development, data governance, service layers, and implementation details for building scalable big‑data solutions.
The Longgong Data Analysis Platform enables Idle Fish to capture, store, and analyze billions of structured product attributes in real time across more than 8,000 categories, using TableStore, MySQL, ODPS, and a distributed scheduler to achieve over 50% query speedup, 80% category coverage, and rapid support for search and recommendation teams.
This article provides a comprehensive technical overview of data middle platforms, covering data aggregation, offline and real‑time development, smart operations, data asset management, governance, service layers, platform implementations, warehouse layering, and key differences between offline and real‑time data warehouses.
This article explains the origins, concepts, and use‑cases of Event Stream Processing (ESP) and Complex Event Processing (CEP), contrasting their handling of ordered event streams versus unordered event clouds, and discusses how both technologies have evolved and are applied in modern real‑time analytics.
ClickHouse, born from Yandex’s Metrica and now a top‑50 open‑source analytics engine, achieves exceptional speed through a vectorized compute engine, column‑store architecture, and an active community, powering real‑time workloads at companies like Tencent Music, Sina, Bilibili, and Suning while introducing features such as column merging, projections, and storage‑compute separation for future scalability.
The article describes how Yuanfudao's data middle platform built a high‑performance OLAP service using the MPP HOLAP engine DorisDB to unify real‑time and batch analytics, meet low‑latency and high‑concurrency requirements, and support diverse education‑industry use cases such as live‑stream monitoring, advertising, and order analytics.
Yuanfudao's data middle platform leverages the MPP database DorisDB to create a unified OLAP system that supports both real‑time and batch analytics, handling millions of queries daily with sub‑second latency while meeting complex business requirements across its education services.
This article describes how JD.com leverages the open‑source MPP analytical database Doris for real‑time and offline OLAP on customer‑service data, covering data ingestion pipelines, dual‑stream high‑availability design, dynamic partition management, multi‑level caching, monitoring with Prometheus‑Grafana, and performance optimizations applied during major sales events.
This article presents a comprehensive design of a real‑time security monitoring and alert platform, detailing challenges in high‑concurrency risk control, an architecture that replaces OLAP polling with scalable compute services, event‑time processing, dynamic thresholding using fbprophet, and practical optimizations with Redis and ClickHouse.
This article details Beike's large‑scale OLAP platform, explaining why Druid was chosen over Kylin, describing the platform's four‑layer architecture, presenting performance and storage benchmarks, and outlining practical improvements to data ingestion, real‑time distinct counting, and cluster stability for high‑concurrency business scenarios.
The article examines the key challenges of real‑time data aggregation—data freshness, timely processing, and result visibility—and surveys common solutions such as timestamp‑based sync, CDC, full and incremental computation, storage formats, and trigger mechanisms.
The article details how Netflix collects massive real‑time device logs, streams them through Kafka into Apache Druid, and uses this high‑performance analytical database to monitor, query, and continuously improve user experience at a scale of over two million events per second.
This article details how KuJiaLe's big data team replaced their legacy ADB and Presto clusters with a DorisDB MPP database, achieving sub‑second query latency, unified real‑time and offline analytics, simplified ETL pipelines, and significant cost savings while supporting billion‑row tables and high‑QPS workloads.
This article details ByteDance's evolution from offline ClickHouse ingestion to a robust real‑time analytics pipeline, covering external transaction handling, risks of direct INSERTs, recommendation and ad‑delivery use cases, Kafka Engine design, multi‑threaded consumption, fault‑tolerance improvements, platform tooling, and future roadmap.
This article provides a detailed learning roadmap for Apache Flink, covering its theoretical background, key research papers, fundamental concepts, core modules, source‑code exploration, real‑time data‑warehouse use cases, event‑driven applications, and emerging trends in the big‑data ecosystem.
This article details JD Search's journey from a Storm‑based pipeline to a Flink‑driven architecture backed by Apache Doris, covering business requirements, technical challenges, design trade‑offs, performance optimizations for massive traffic spikes, and future plans for their real‑time OLAP data warehouse.
This article explains how to implement dynamic data partitioning in Apache Flink using a fraud‑detection demo, covering the system architecture, rule‑driven runtime reconfiguration, custom ProcessFunction code, and the underlying key‑by logic that enables flexible, real‑time stream processing.
This article presents Meituan's use of Flink to enable incremental data warehouse production, covering the warehouse architecture, streaming data integration evolution, real-time OLAP applications, platform design, and future directions for unified stream‑batch processing.
This article details how Beike built a comprehensive Data Management Platform (DMP) that integrates user behavior and business data across multiple apps, outlines its five‑layer architecture, discusses data collection, processing, storage, real‑time profiling, and presents performance results and future optimization directions.
This article explains how JD’s Energy Management Platform uses ClickHouse as an MPP‑based OLAP engine to ingest, store, and provide multi‑dimensional real‑time analytics on energy consumption data, covering architecture decisions, data pipelines, replication, sharding, and a generic query interface.
The article explains ClickHouse’s high‑performance columnar OLAP design, its vectorized execution, sparse primary‑key indexes and MergeTree engines, contrasts it with ROLAP/MOLAP approaches, and details Youzan’s large‑scale deployment—including dual‑replica clusters, ingestion pipelines, routing architecture, current challenges, and future container‑based expansion plans.
In 2020, Apache Flink surged to become the most active Apache project, releasing three major versions that advanced its unified stream‑batch engine, introduced cloud‑native K8s support, expanded AI capabilities with PyFlink, and fostered a thriving Chinese community, solidifying its role as the de‑facto standard for real‑time computing.
This article provides a comprehensive introduction to stream processing fundamentals by distinguishing between bounded and unbounded datasets, clarifying the critical differences between event time and processing time, and exploring various windowing strategies to demonstrate how modern distributed systems efficiently handle continuous data flows.
This article details how Ctrip's senior engineering manager leveraged ClickHouse to build a high‑availability, sub‑second response data platform handling nearly 700 billion rows, describing the motivations, architecture, data synchronization processes, performance gains, challenges, and practical recommendations for large‑scale analytics.
This article provides a comprehensive overview of data middle platform concepts, covering data aggregation, collection tools, development modules, job scheduling, baseline control, heterogeneous storage, permission management, real‑time and offline processing, governance, services, and implementation details for building robust big‑data solutions.
This article explains how Alibaba's traffic analysis platform evolved to require real‑time analytics, outlines the business background, data model and metric system, discusses the difficulties of big‑data processing, and describes why ClickHouse was chosen and how its features solve those challenges.
This article summarizes a JD Cloud‑TiDB live webcast, explaining why JD Cloud chose TiDB, detailing its distributed architecture (TiDB Server, TiKV storage, PD), highlighting features such as horizontal scaling, strong consistency, real‑time analytics, backup, migration, and showcasing multiple enterprise migration case studies that demonstrate TiDB’s ability to overcome traditional database bottlenecks.
Alibaba’s 2020 Double‑11 achieved unprecedented real‑time processing of 4 billion records per second and 7 TB of data per second using Flink, showcasing the stability, performance and efficiency of its stream‑batch unified architecture across diverse business scenarios.
This article walks through the evolution of big‑data systems to modern stream processing, explains core Flink concepts such as state, checkpoints, event‑time and windowing, and details Kuaishou’s real‑time UV calculation and fast‑failover techniques for high‑availability streaming jobs.
This article explains how to build a real‑time click‑stream data warehouse using Flink for stream processing and ClickHouse for near‑real‑time OLAP, covering click‑stream characteristics, dimensional modeling, layered warehouse design, async dimension joins, sink implementation, and data rebalancing strategies.
Lyft transformed its legacy data pipeline by designing a cloud‑native, Flink‑based near real‑time analytics platform that ingests billions of events, writes Parquet files to S3, leverages Presto for interactive queries, and implements multi‑stage non‑blocking ETL, fault‑tolerant back‑fill, and extensive performance optimizations.
