This article walks you through the fundamentals of data governance, explains metadata management concepts, compares traditional tools with DataHub, and provides a step‑by‑step tutorial for installing Docker, Python, and DataHub 0.8.20 on CentOS 7, ingesting MySQL metadata, and exploring the UI.
The second virtual research meeting of China’s Data Science Curriculum Group gathered nearly a hundred educators and industry partners in Beijing to discuss new models for big‑data course design, curriculum construction, industry‑academia collaboration, and digital teaching platforms across multiple universities.
The article presents practical ODPS SQL optimization strategies—including explicit column selection, partition limiting, multi‑insert, proper handling of nulls, join‑type choices, map‑join and skew hints, bucketed tables, and tuned task parameters—illustrated with three real‑world cases that dramatically cut execution time and resource usage.
Lancer, Bilibili’s real‑time streaming backbone, has evolved from a monolithic Flume pipeline to a log‑id‑isolated, Kubernetes‑native architecture where Go edge agents feed synchronous Kafka‑proxied gateways into per‑logid topics processed by dedicated Flink‑SQL jobs, delivering exactly‑once, back‑pressured, highly scalable data ingestion for billions of daily requests.
The DataFun Summit 2022 featured a Data Lake & Warehouse forum with expert talks on PALO, ByteDance LAS, Iceberg at Huawei, and Presto‑Alluxio acceleration, providing detailed technical outlines, speaker backgrounds, and audience takeaways for modern big‑data architectures.
The article explains the fundamental differences between informationization and digitalization, outlines how enterprises can bridge the gap through data‑driven strategies, and provides practical frameworks and case studies such as Netflix and Huawei to guide traditional manufacturers in successful digital transformation.
This article describes Bilibili's offline multi‑datacenter architecture, explaining why a scale‑out approach was chosen over scale‑up, and detailing the unit‑based design, job placement, data replication, routing, versioning, bandwidth throttling, traffic analysis, and the operational results and future directions.
This article details ZhongAn Insurance's digital transformation through its 4633 data‑centric framework, the architecture of its JiZhi data platform, the challenges of its original ClickHouse‑based real‑time warehouse, and how migrating to StarRocks improved performance, scalability, and operational efficiency across advertising and insurance use cases.
Shopee tackled its data‑system bottlenecks by customizing Apache Hudi to provide unified stream‑batch integration, efficient state‑detail snapshots, and low‑latency wide‑table generation, using CDC‑based bootstrapping, COW/MOR tables, savepoints and partial updates, which cut latency to ten minutes, lowered resource use, and yielded several community‑backed enhancements.
This article dissects why a well‑tuned Kafka cluster can process up to 20 million messages per second, examining producer batching and custom protocols, broker page‑cache, file layout and zero‑copy techniques, as well as consumer group strategies that together unlock its high throughput.
The article introduces Tencent's big‑data platform philosophy and overall architecture, detailing three generations of evolution from offline Hadoop‑based processing to real‑time Spark/Storm integration and finally AI‑driven machine‑learning platforms, while also highlighting the team, book publication, and a related giveaway event.
This article provides a comprehensive overview of Apache Kafka, explaining its distributed message‑queue architecture, the role of topics and partitions, producer and consumer workflows, leader election, offset management, consumer‑group rebalancing, delivery semantics, transaction processing, file organization, and key configuration settings.
This article presents the design, implementation, and evaluation of CloudRCA, an intelligent root cause analysis framework for Alibaba Cloud's big‑data computing services, detailing challenges such as heterogeneous data, sample imbalance, and real‑time constraints, and describing the multi‑stage data processing, hierarchical Bayesian modeling, and deployment results that reduce MTTR by 20%.
This article explains the design and implementation of a unified data security control platform that protects user privacy and corporate data across multiple big‑data components (Hive, Hetu, GaussDB) by integrating Apache Ranger, custom authorization APIs, asynchronous processing, distributed locking, and SDK‑based authentication to achieve fine‑grained, one‑stop permission management.
The article explains data replication concepts, processes, and technologies across storage hardware, operating system, and database layers, outlines synchronous, asynchronous, and hybrid methods, discusses industry applications, trends such as hardware‑software decoupling, cloud replication, and big‑data real‑time copying, and highlights challenges and future directions.
Baidu Intelligent Cloud's Flow Log product provides real‑time, high‑throughput network flow collection, visualization, and analysis for VPC, dedicated line, and NAT gateways, enabling fault diagnosis, cost allocation, elephant‑flow management, and security inspection across ultra‑large scale cloud environments.
This article explains ByteDance's end‑to‑end data‑point (埋点) validation system, covering its technical challenges—usability, accuracy, real‑time visibility, stability, and extensibility—along with SDK integration, QR‑code workflow, JSON‑Schema verification, push‑service architecture, SLA metrics, and future automation plans.
This article surveys twenty data‑visualization tools—covering entry‑level options like Excel, online JavaScript libraries such as D3 and Google Chart API, interactive GUI utilities, map frameworks, advanced desktop environments, and expert‑grade platforms like R, Weka and Gephi—highlighting their key features, formats supported and typical use cases.
The Apache DolphinScheduler 3.0.0 release on August 10, 2022 introduces a faster UI, stronger data‑quality guarantees, modernized design, easier maintenance, AWS support, service splitting, and native Flink task support, accompanied by detailed code examples and download links.
This article explains the purpose, sources, and manifestations of pressure in Alluxio, describes its built‑in stress testing framework, outlines how to run and configure stress tools, and provides guidance on result calculation, reporting, common issues, and debugging for effective performance evaluation.
This article presents a comprehensive overview of Taobao's data model governance, detailing the background and problems of the current data architecture, analyzing root causes, proposing a structured governance framework with DataWorks automation, and outlining future plans to improve efficiency, standardization, and product tooling.
The DeWu Reach Strategy platform uses a task‑strategy‑action model and an HBase‑backed buffer pool that temporarily stores billions of user records, enabling large‑scale algorithmic push, AB testing, and dynamic horizontal scaling while ensuring even data distribution and low‑latency processing.
This article presents an overview of data governance concepts, common enterprise challenges, and Alibaba's comprehensive data governance framework, covering theory, demand layers, practical solutions for stability, quality, standards, security, cost control, and the supporting platforms and operational practices.
This article presents a comprehensive Python-based analysis of a large game dataset (2.29 million records, 109 fields), covering user registration trends, payment rates, ARPU/ARPPU calculations, level‑based spending behavior, and consumption patterns of resources and acceleration items, with visualizations and actionable conclusions.
This article explains how ByteDance’s event‑tracking (埋点) data flow handles billions of events per second using Flink‑based real‑time ETL, dynamic rule engines, data sharding, and multi‑datacenter disaster‑recovery to ensure stability, low latency, and cost‑effective processing for diverse downstream services.
This article outlines the four major storage challenges of the ABC era—massive scale, cost efficiency, stability, and diversity—and explains how Baidu’s Canghai storage suite, including BOS, CDS, CFS, PFS, RapidFS, CloudFlow, and storage gateways, addresses each through multi‑cloud migration, tiered lifecycle management, and robust disaster‑recovery solutions.
This article explains why event‑tracking (埋点) governance is essential, outlines the methodology and practice of full‑link tracking management, and introduces the one‑stop tracking platform with its innovative features such as standardized processes, verification tools, real‑time dashboards, cross‑platform data unification, and future roadmap.
This article explains the eight built‑in Flink partitioners, their distribution strategies, key implementation details, and provides Java code examples illustrating how each partitioner selects downstream channels and determines pointwise or all‑to‑all distribution.
Pinterest’s Monarch batch‑processing platform, built on over 17 k YARN nodes in AWS, was upgraded from Hadoop 2.7.1 to 2.10.0 using a phased, cluster‑by‑cluster strategy that balanced minimal downtime, extensive validation, and custom patches to handle compatibility and dependency issues.
The talk outlines KuaiKan's data governance journey, describing the rapid business growth challenges, the three‑step logical closed‑loop framework, practical experiences in business scope management, data asset governance, collaboration techniques, and future outlook, highlighting evaluation metrics and ongoing improvements.
This article explains how Alibaba leverages Flink to handle over 40 billion events per second across all business units, detailing risk‑control concepts, rule types, architectural stages, resource tuning, dynamic CEP, shared computing, and the FY23 roadmap for large‑scale streaming risk management.
This tutorial demonstrates how to accelerate processing of a multi‑million‑row CSV dataset by splitting the work into sub‑tasks and applying Python's multiprocessing, joblib, and tqdm libraries for serial, parallel, and batch processing, showing significant speed‑ups and best‑practice code snippets.
This article introduces the basic concepts of knowledge graphs, explores their research dimensions across knowledge engineering, natural language processing, databases and machine learning, discusses graph database storage models and their integration with artificial intelligence and big data, and presents related projects and real‑world case studies.
This article presents Huace Data Science's practical approaches to digital supply‑chain finance for SMEs, detailing challenges of cross‑industry data, the SME engine for authentic business assessment, graph‑based fraud detection, and quantum‑inspired feature‑engineering methods that enhance credit‑risk models.
This article examines the evolution of big‑data architectures, identifies the latency and maintenance issues of classic Lambda designs, and presents a hybrid Lambda‑Kappa solution that unifies streaming and batch processing to achieve minute‑level data freshness and second‑level query latency while reducing development cost.
This article introduces graph database fundamentals, explains why graph databases are needed, outlines core storage goals such as index‑free adjacency, compares array, linked‑list and LSM‑tree storage schemes, and presents the design, performance advantages, and real‑world applications of the Galaxybase distributed graph database.
This article provides a comprehensive overview of the big data ecosystem, detailing nine core technology categories—from data collection and storage to computation, analysis, scheduling, and underlying infrastructure—along with tool comparisons, selection guidelines to help readers quickly build a complete big data knowledge system.
This article provides a comprehensive overview of Alibaba Cloud MaxCompute, covering its product features, architecture, ecosystem integrations, and in‑depth data security mechanisms such as authentication, RAM roles, access control policies, label‑based security, project protection, audit logging, encryption, backup, disaster recovery, and the complementary DataWorks security capabilities.
This article explains Doris's fundamental concepts such as rows, columns, tablets, and partitions, provides guidelines for column definition, partitioning and bucketing strategies, and details table creation syntax and property settings for optimal big‑data storage and query performance.
The article examines the rise of smart pens, point‑matrix technology, and other AI‑driven monitoring tools in Chinese schools, detailing how they record handwriting, emotions, screen activity, and even biometric data, while raising privacy concerns and highlighting the massive market for educational surveillance.
This article outlines Snowball's data warehouse background, compares major modeling approaches such as ER, dimensional, DataVault and Anchor models, describes the current challenges of their dimensional model, and details the OneData methodology—including OneModel, OneID, and OneService—along with its practical implementation, results, and future plans.
The presentation details how Amazon Web Services’ intelligent data lake architecture integrates big data and machine learning to overcome marketing challenges, improve data governance, and provide scalable, real‑time analytics for personalized, data‑driven marketing across enterprises.
A recent IDC report reveals that Alibaba Cloud captured 14.9 billion yuan in revenue, securing the top spot in China’s big data platform public‑cloud market in 2021, driven by rapid 53.8 % growth and emerging technologies such as real‑time data warehouses, lake‑house integration, streaming‑batch convergence, and AI‑enabled analytics.
Huolala’s data‑engineer Yang Qiuji shares how the company’s OLAP platform progressed from Druid (OLAP 1.0) to ClickHouse (OLAP 2.0) and finally to Apache Doris (OLAP 3.0), detailing business drivers, technical evaluations, POC results, stability measures, and future roadmap.
This article explains how Neutrino’s SerializableProvider API enables passing final classes, managing mutable object state, and supporting Spark checkpoint recovery through dependency injection, while also showing practical code patterns and injection of core Spark components.
This article provides a detailed overview of DataX, covering its purpose, framework design, core architecture, scheduling process, practical examples of MySQL-to-MySQL synchronization, step‑by‑step installation and configuration of DataX‑WEB, UI usage, routing strategies, task types, and advanced task building techniques.
This article explains Kafka's core concepts—including topics, partitions and replicas, log segment storage, leader‑follower mechanics, consumer groups, network threading model, zero‑copy I/O, and the essential role of Zookeeper for broker, topic, consumer, and offset management—providing a comprehensive overview for developers and architects.
The article explains how Spark Streaming's checkpoint mechanism works, detailing the four-step process—from setting the checkpoint directory to writing RDD data and finalizing the checkpoint—highlighting its role in ensuring fault‑tolerant, fast recovery for real‑time recommendation feature pipelines.
This article presents Tencent's PCG data platform evolution, detailing the challenges of integrating multiple business groups, the design of a unified big‑data architecture, real‑time and batch processing pipelines, MQ and ATTA systems, and comprehensive operational practices for reliability and scalability.
China Telecom is creating a state‑backed “national cloud” by partnering with multiple central‑enterprise investors, consolidating resources, accelerating indigenous cloud technology, and setting ambitious infrastructure targets, while similar initiatives emerge worldwide in the US, Russia, India, France and Italy.
This article details Bilibili's evolution from a Hadoop‑based offline platform to a Spark‑driven architecture, covering the Hive‑to‑Spark migration, automated SQL conversion, result validation, stability enhancements, performance tuning, meta‑store federation, and future directions for large‑scale data processing.
Sugar BI, Baidu Cloud’s AI‑powered business intelligence platform, lets users create professional, zero‑code dashboards in minutes by connecting to 30+ data sources, leveraging Apache ECharts, intelligent chart recommendation, and natural‑language voice interaction to deliver automated analysis, visualization, and predictive insights.
This article provides a comprehensive overview of data lakes, explaining their definition, key characteristics, architectural evolution, and detailed comparisons of major cloud providers' solutions, while also presenting typical use cases, construction processes, and future development directions for this emerging big‑data infrastructure.
This article presents a comprehensive overview of NetEase's log collection and transmission platform, detailing its evolution from 2011 to the current Datastream‑NG architecture, the system's design goals, core component optimizations, operational monitoring, and future plans for intelligent scaling and diagnostics.
This article explains Baidu Intelligent Cloud’s data lake acceleration solution, covering the evolution of big‑data technologies, the benefits and challenges of compute‑storage separation, the architecture of BOS object storage, and the native hierarchical namespace and RapidFS cache mechanisms that boost performance and reduce costs.
This article explores data visualization fundamentals, compares scientific, information, and analytical visualization, reviews popular frontend libraries like ECharts and AntV/G2, showcases real-world case studies, and details technical choices for building interactive charts and graph‑based analytics in modern applications.
The author shares a candid account of transitioning from a non‑data role to confronting data‑centric bottlenecks, describing the current state of data projects, common pitfalls, and practical thoughts on simplifying data governance within limited resources and budget constraints.
This article provides a comprehensive tutorial on setting up Flink 1.11 with Iceberg 0.11.1, creating Hive catalogs, building databases and tables, inserting data, and exploring Iceberg components, file structures, partitioned tables, execution plans, and programmatic access via Scala.
This article shares FenbeiTong's experience in building a big data platform, covering company background, data construction challenges, technology selection, architecture design, implementation details, data modeling tools, and real-world application scenarios such as CDP and CEM, offering practical insights for similar enterprises.
The Red Hat 2019 Enterprise Open Source Survey summarizes the most widely adopted open‑source projects in enterprises, covering web servers, big‑data frameworks, cloud platforms, distributed storage, operating systems, databases, development tools, and middleware, and highlights their strategic importance for modern IT infrastructure.
This talk presents Shuhe Technology’s feature platform, detailing its four‑layer architecture, feature storage services, stream‑batch integrated processing, event‑center design, consistency models, and four model‑strategy invocation schemes, illustrating data flows from MySQL through Sqoop, Kafka, Flink, HBase and ClickHouse.
This report details Alibaba’s large‑scale data model governance initiative for the DaTao ecosystem, analyzing current data issues such as naming inconsistencies, low reuse, and application‑layer inefficiencies, and presents a comprehensive solution—including a model evaluation system, DataWorks co‑development, intelligent modeling, data map enhancements, and future roadmap—to improve data health, reduce costs, and increase operational efficiency.
The article uses an acorn‑moving analogy to highlight latency and traceability challenges in enterprise data warehouses, then explains offline versus real‑time approaches, compares Lambda and Kappa architectures, discusses Iceberg integration, and shares a detailed e‑commerce real‑time warehouse case study with optimization tips.
This article explains the different join types supported by Apache Flink—including regular, interval, temporal, and lookup joins—provides SQL examples, details how the Flink optimizer transforms logical plans into efficient physical plans, and describes the underlying code generation and execution mechanisms.
This article summarizes Guo Jinshi's presentation on Taobao's data model governance, covering the current data landscape, identified problems, analysis of root causes, proposed governance solutions—including DataWorks intelligent modeling—and future plans, while also providing a Q&A session on practical implementation.
The article explains how space‑efficient probabilistic structures such as BloomFilter and Count‑min Sketch enable large‑scale data deduplication, join pruning, real‑time idempotent filtering, and approximate top‑K analytics by trading modest accuracy loss for dramatically reduced storage and faster computation.
This article provides a comprehensive tutorial on preparing the environment, installing DataX and DataX‑web, configuring MySQL, JDK, Maven, and Python, deploying the services on Linux, and using the web UI to create data sources, build JSON jobs, monitor execution, and manage users.
This article explains Apache Doris’s architecture, introduces the Hudi data‑lake format, compares Lambda and Kappa approaches, and details the design and implementation of Doris’s Hudi external table support, including practical steps, code examples, and future roadmap.
This talk by Zhongan Insurance’s Data Senior Director Shi Xingtian outlines the company’s digital transformation, detailing the 4633 framework, the real-time data warehouse architecture, the migration from ClickHouse to StarRocks, and how these technologies support fine‑grained, intelligent financial operations and advertising analytics.
This article presents a comprehensive overview of deploying and running graph learning algorithms—both inductive and transductive—on the secure Spark cluster, covering framework choices, data sampling strategies, distributed training techniques, model evaluation metrics, and future directions.
The talk traces the evolution of API Gateway architectures and the engineering practices—design patterns, deployment strategies, and operational considerations—required for scalable, reliable services, then demonstrates how ClickHouse can be leveraged for massive data workloads, highlighting practical scenarios, performance optimizations, and key lessons learned.
This article chronicles NetEase Games' evolution of its real‑time StreamflySQL platform, detailing the transition from a client‑side Flink SQL implementation to a server‑side architecture powered by SQL Gateway, and discusses the motivations, design choices, challenges, and performance improvements achieved.
37 Mobile Games, a leading mobile game publisher, migrated its user‑profile analytics from a Hadoop‑Hudi‑Kafka‑Hive‑Flink stack to StarRocks, achieving sub‑second query latency on billion‑row tables, simplifying operations, reducing storage costs, and enabling real‑time data sync, as detailed in this technical case study.
This article describes Huya's large‑scale offline‑online mixed deployment, detailing the low resource‑utilization problems, the time‑sharing and elastic scheduling solutions, the containerized architecture, multi‑datacenter isolation, heterogeneous resource handling, stability safeguards, and the resulting performance improvements and future directions.
This article shares practical experiences and foundational concepts for creating data‑visualization dashboards, covering screen types, design principles, characteristics, audience analysis, and the broader role of visualization in turning massive data into actionable insights while enhancing human cognition.
This article details how Zuoyebang migrated its Hive‑based offline data warehouse to DeltaLake, addressing latency, incremental updates, and query performance through stream‑to‑batch processing, dynamic partition pruning, and Z‑order optimization, resulting in faster data readiness and analyst queries.
This article provides a comprehensive overview of Kafka, covering its fundamental concepts such as producers and consumers, topics and consumer groups, partitions and ordering, as well as the cluster architecture involving ZooKeeper, replication, and leader‑follower mechanisms, illustrated with diagrams.
This article details the design and implementation of JD Retail’s product selection platform, covering its business background, core data retrieval capabilities, domain model, system architecture—including frontend configurability, backend query engine, ClickHouse indexing, and both offline and real-time data processing pipelines.
This article explains how ByteDance’s dynamic data exploration tool improves data quality assurance by replacing time‑consuming SQL validation with real‑time, sample‑based profiling, detailing its problem background, core features, technical architecture, front‑end rendering techniques, operation‑stack management, and future enhancements.
This article thoroughly examines Kafka's broker‑side network architecture, tracing its evolution from a simple sequential model to a high‑performance, event‑driven Reactor design using Java NIO, and provides practical tuning guidance for achieving optimal throughput and latency.
This article explains what Elasticsearch is, how it solves database acceleration, log observability, and data analysis problems, details its core components and underlying engine features, compares its strengths and weaknesses, and presents classic application scenarios and a real‑world case study integrating Elasticsearch with Flink for large‑scale log analytics.
The presentation details the motivations, goals, and architectural redesign of Apache SeaTunnel (Incubating) to decouple its Source and Sink APIs from underlying engines, introducing unified APIs, version‑agnostic connectors, and enhanced support for Spark and Flink in both batch and streaming scenarios.
The article provides an in‑depth overview of Apache Hudi 0.11.0, covering its new multi‑level index design, Spark SQL enhancements, Flink integration improvements, and additional performance and usability features aimed at boosting read/write efficiency in large‑scale data lake environments.
Xinwang Bank leverages Apache DolphinScheduler to handle over 9,000 daily task instances across real‑time, near‑real‑time, and offline batch scenarios, detailing background, application scenarios, optimizations, workflow improvements, import/export enhancements, alert system upgrades, and future plans to expand data‑ops capabilities.
This article explains Bilibili's lake‑warehouse integrated architecture, describing how Iceberg, MagnuS, Trino, and Alluxio are used to achieve flexible data storage, high‑performance query acceleration, and automated indexing through Z‑Order, Hilbert curve, Bloom filter, and advanced BitMap techniques.
This article provides a comprehensive introduction to Elasticsearch, covering data types, Lucene fundamentals, cluster architecture, node roles, shard and replica mechanisms, mapping, installation, health monitoring, indexing principles, storage strategies, refresh and translog handling, segment merging, performance tuning, and JVM optimization for large‑scale search applications.
This article walks through using Apache Spark for large‑scale GBDT training, covering the challenges of massive data, Spark deployment, PySpark code examples, differences from Pandas, feature engineering, mmlspark installation, early‑stopping tricks, performance bottlenecks, and a systematic evaluation of alternative frameworks.
This article provides a detailed overview of Elasticsearch, covering its data model, Lucene foundation, cluster architecture, shard and replica mechanisms, index mapping, installation steps, health monitoring, write and storage processes, segment management, and performance tuning techniques for large‑scale search applications.
This article explains why traditional mysqldump and file‑based methods struggle with massive tables, introduces Alibaba DataX as a high‑performance offline data integration tool, details its architecture, and provides comprehensive installation and configuration steps for full and incremental MySQL‑to‑MySQL synchronization using JSON job files.
The article outlines how the COVID‑19‑driven shift to remote work accelerated digitalization, describes the rapid growth of the digital economy, explains the two‑step process of industry digitization and digital industrialization, and highlights the strategic role of AI, cloud computing, big data, 5G and digital twins in reshaping enterprises across sectors.
From data ingestion to real‑time analytics, this guide breaks down the essential layers of a typical big‑data platform—covering collection methods, HDFS storage, Hive/Spark analysis, data sharing mechanisms, application use‑cases, streaming with Spark Streaming, and the need for robust scheduling and monitoring.
This article details why Microvision adopted Apache Iceberg, how it replaces parts of their Lambda‑architecture data pipeline, the real‑time and offline use cases, table‑maintenance practices such as snapshot cleanup and small‑file merging, and lessons learned from the implementation.
This article explains common Kafka message‑loss and duplicate‑consumption issues, introduces Alibaba Cloud's fully managed Kafka Retrieval Component, and provides step‑by‑step guidance—including enabling the service, using Tablestore for multi‑index and SQL searches—to help engineers quickly locate and verify missing or duplicated messages.
This article explains how Spark reads data from Apache Iceberg tables by parsing snapshots and manifest files into DataFile objects, creates Batch and InputPartition objects, uses readers to materialize InternalRows, and then demonstrates how Iceberg's RewriteDataFilesAction can merge tiny Parquet files into larger ones through Spark‑driven tasks.
This article explores the value and evolution of predictive maintenance (PdM), outlines common technical approaches—including signal processing, mechanism + big‑data, digital twin, and AI—examines time‑series database choices such as MatrixDB, presents case studies and practical insights, and concludes with reflections on industrial digital transformation.
This article presents a comprehensive overview of how Amazon EMR Serverless leverages serverless technology to simplify, scale, and cost‑optimize big data analytics, covering the evolution of serverless services, the intelligent lakehouse architecture, core concepts, key benefits, common use cases, and available documentation.
DataFun's 2022 summit brings together leading experts from academia and industry to discuss privacy computing, federated learning, secure data sharing, and their applications across finance, healthcare, telecom, and blockchain, offering insights into technologies, standards, and real-world implementations that enable data utility while protecting privacy.
This article explains Apache Iceberg’s architecture, core capabilities such as time‑travel, fast scans, delete handling, and schema evolution, and provides a step‑by‑step guide for configuring Flink to use Iceberg with Hive and Hadoop catalogs, including DDL commands and streaming queries.
The article describes how Ctrip built a unified data service platform that standardizes API development, leverages multiple storage engines, introduces token‑based security, Sentinel rate‑limiting, caching, and automatic contract generation to dramatically cut development cycles and improve reliability for big‑data workloads.
This article details Hulu's five‑year journey from Hadoop 2.6 to 3.3.2, covering major feature evolutions, the original cluster architecture, a comprehensive upgrade plan, compatibility challenges across HDFS, YARN, Hive, Spark and Flink, and the testing and rollout strategies that ensured a smooth migration.
