Author

Big Data Technology & Architecture

Wang Zhiwu, a big data expert, dedicated to sharing big data technology.

1.0k

Articles

Likes

426

Views

Comments

Latest from Big Data Technology & Architecture

100 recent articles max

Big Data Technology & Architecture

Dec 12, 2024 · Big Data

Understanding Time Travel and Snapshot Retention in Lake Frameworks (Hudi & Paimon)

This article explains how lake frameworks like Hudi and Paimon implement Time Travel by recording older data versions, the snapshot retention policies that limit historical data access, and practical recommendations for managing snapshots and consumption patterns to reduce storage costs in large‑scale data warehouses.

Big DataHudiPaimon

0 likes · 7 min read

Understanding Time Travel and Snapshot Retention in Lake Frameworks (Hudi & Paimon)

Big Data Technology & Architecture

Dec 9, 2024 · Big Data

Understanding Flink’s Exactly-Once Semantics and Its Relation to Deduplication

This article explains what Flink’s Exactly‑Once semantics actually guarantee, why it does not mean each event is processed only once, how checkpointing and two‑phase commit sinks enable end‑to‑end exactly‑once, and the three safeguards needed for true exactly‑once computation.

Big DataDeduplicationExactly-Once

0 likes · 5 min read

Understanding Flink’s Exactly-Once Semantics and Its Relation to Deduplication

Big Data Technology & Architecture

Dec 2, 2024 · Big Data

Optimizing Primary‑Key and Append‑Scalable Tables in Paimon with Flink

This guide explains how to optimize Paimon primary‑key and Append‑Scalable tables in Flink by adjusting sink and source parallelism, checkpoint intervals, making small‑file merges fully asynchronous, changing file formats, and applying ordering strategies to improve both write and read performance.

BatchBig DataFlink

0 likes · 6 min read

Optimizing Primary‑Key and Append‑Scalable Tables in Paimon with Flink

Big Data Technology & Architecture

Nov 26, 2024 · Big Data

Understanding Full GC, Data Skew, and Parallelism in Flink Tasks

This article explains how to monitor and interpret Full GC in Flink TaskManagers, detect and address data skew through proper data distribution and parallelism settings, and recommends aligning consumer parallelism with Kafka partitions, while also providing practical tips for using tools like Prometheus and Arthas.

Data SkewFlinkFull GC

0 likes · 6 min read

Understanding Full GC, Data Skew, and Parallelism in Flink Tasks

Big Data Technology & Architecture

Nov 25, 2024 · Big Data

Tencent Real-Time Lakehouse Architecture and Intelligent Optimization Practices

This article presents Tencent's real‑time lakehouse architecture, detailing its three‑layer design of compute, management and storage, and explains the six components of the Intelligent Optimization Service—including Compaction, Index, Clustering, and AutoEngine—along with scenario‑based capabilities, migration strategies, and future optimization directions.

Big DataTencentdata optimization

0 likes · 11 min read

Tencent Real-Time Lakehouse Architecture and Intelligent Optimization Practices

Big Data Technology & Architecture

Nov 18, 2024 · Databases

How Useful Is Good SQL Writing and How Deep Should You Master It?

The article argues that while solid SQL skills are essential for data development and interview preparation, investing excessive effort to master advanced SQL yields diminishing returns, especially as AI tools automate routine queries, and recommends focusing on newer, more impactful technologies instead.

AIGCDatabase SkillsInterview preparation

0 likes · 6 min read

How Useful Is Good SQL Writing and How Deep Should You Master It?

Big Data Technology & Architecture

Nov 12, 2024 · Big Data

Adaptive Query Execution (AQE) in Apache Spark 4.0: A Revolution in Query Optimization

This article explains how Adaptive Query Execution (AQE) in Apache Spark 4.0 dynamically optimizes query plans through features such as join reordering, partition pruning, skew handling and coalescing, delivering significant performance gains, resource efficiency and reduced manual tuning across real‑world big‑data workloads.

Adaptive Query ExecutionApache SparkBig Data

0 likes · 13 min read

Adaptive Query Execution (AQE) in Apache Spark 4.0: A Revolution in Query Optimization

Big Data Technology & Architecture

Nov 7, 2024 · Big Data

Douyin Group's Data Management Strategies: Enhancing Metric Stability and Reusability

This article outlines Douyin Group's approach to handling petabyte‑scale data, addressing metric inconsistencies, and improving data product agility through a four‑layer Volcano Engine platform, systematic indicator production‑management‑consumption cycles, organizational design, automation, and future plans for large‑model‑driven metric splitting.

AnalyticsBig DataData Management

0 likes · 20 min read

Douyin Group's Data Management Strategies: Enhancing Metric Stability and Reusability

Big Data Technology & Architecture

Nov 4, 2024 · Databases

Detailed Analysis of Doris SQL Execution Process: Optimizer, Scheduler, and Executor

This article provides a comprehensive walkthrough of Doris's SQL execution pipeline, covering the query optimizer's parsing, rewriting, and plan generation, the scheduler's fragment distribution, and the executor's fragment processing, including code examples of expression rewrite rules, join strategies, and data flow between FE and BE nodes.

Distributed ExecutionQuery OptimizerSQL

0 likes · 30 min read

Detailed Analysis of Doris SQL Execution Process: Optimizer, Scheduler, and Executor

Big Data Technology & Architecture

Nov 1, 2024 · Big Data

Real‑Time Lakehouse Architecture at Ximalaya Live: Leveraging Flink, Paimon, and StarRocks

This article details Ximalaya Live's transition from an offline‑centric data warehouse to a real‑time lakehouse using Flink, Paimon, and StarRocks, covering business background, architectural challenges, technology evaluation, implementation steps, encountered issues, performance gains, and future expansion plans.

FlinkLakehousePaimon

0 likes · 12 min read

Real‑Time Lakehouse Architecture at Ximalaya Live: Leveraging Flink, Paimon, and StarRocks