1 views collected around this technical thread.
This article explains the origins of big‑data technologies, details the architecture and read/write mechanisms of Hadoop's HDFS, describes the MapReduce programming model, and provides complete Java code examples for a simple distributed file‑processing job using Maven dependencies.
This article explains what constitutes a small file in HDFS, why they harm memory, compute and cluster load, outlines common sources such as data sources, streaming and dynamic partitioning, and provides detailed Hive and Spark solutions—including CombineHiveInputFormat, merge parameters, distribute by, and custom Spark extensions—to efficiently merge small files and improve job performance.
This article explains Hadoop’s core concepts using a library analogy, details HDFS storage and MapReduce processing, provides complete Java implementations for a word‑count job with support for text, CSV, and JSON inputs, and discusses extensibility and performance optimizations such as combiners and custom partitioners.
This article introduces PowerJob, a young yet mature distributed task scheduling framework, explains why it was chosen, details its architecture, high‑availability design, deployment steps, and demonstrates various job types—including standalone, broadcast, map, and MapReduce—along with CRON, fixed‑rate, and fixed‑delay scheduling configurations.
The paper presents a centralized online batch‑processing framework for large‑scale promotion systems, where applications integrate via an SDK, a task‑center schedules and dispatches sub‑tasks through RocketMQ to Dubbo‑enabled containers, employing MapReduce‑style splitting, Guava rate‑limiting, heartbeat health checks, and has successfully handled over 1.3 million tasks during Double‑11.
This article introduces the PowerJob distributed task framework, explains why it was chosen, details its architecture and high‑availability design, demonstrates various job types—including standalone, broadcast, map, and map‑reduce—with Java code examples, and covers scheduling options such as CRON, fixed‑rate, and fixed‑delay execution.
The article explains how to handle business scenarios that require splitting large data sets into concurrent I/O requests and sequential aggregation by presenting a Go‑based chunk processing framework with map and reduce functions, configurable concurrency, and example code.
PowerJob is a modern distributed job scheduling framework that addresses the limitations of Quartz, XXL‑Job and SchedulerX by offering a web UI, rich scheduling strategies, DAG workflow support, lock‑free high‑performance scheduling, multiple processor types and step‑by‑step quick‑start instructions for developers.
PowerJob is a third‑generation distributed job scheduler that adds workflow orchestration, map‑reduce style computation and rich execution modes to traditional CRON‑based scheduling, and this guide explains its advantages, core features, architecture, and provides step‑by‑step instructions with code samples to get started quickly.
This comprehensive article explains big data concepts, definitions from Gartner and IBM, real‑world use cases, the Hadoop ecosystem architecture, and detailed introductions to HDFS, MapReduce, YARN, Hive, and HBase, including practical examples and shell commands.
This article provides a detailed, step-by-step tutorial for installing Hadoop 2.9.2, configuring environment variables, editing XML configuration files, formatting the NameNode, starting HDFS and YARN services, testing the cluster, and setting up the MapReduce history server on a three‑node Linux environment.
This article presents a detailed interview-style walkthrough covering Hadoop cluster setup, HDFS components, MapReduce workflow, YARN advantages, Spark fundamentals, Kafka replication, Hive table types, and related big‑data concepts, providing concise explanations and practical insights for data engineers.
This article explains the four locations where java.lang.OutOfMemoryError can occur in Hadoop's MapReduce framework—client, ApplicationMaster, Map, and Reduce phases—and provides configuration adjustments and best‑practice solutions to mitigate each type of OOM issue.
This article provides a thorough overview of Apache Spark, covering its origins, comparison with MapReduce, core concepts such as RDD, DAG, Jobs, Stages, and Tasks, the submission process, Web UI, and detailed performance tuning techniques including data skew mitigation.
This article explains how to efficiently move large volumes of data from Hive to HBase by leveraging HBase's bulkload mechanism, detailing the original MapReduce workflow, its performance bottlenecks, and a rewritten Spark‑based solution that simplifies ETL, improves partitioning, and achieves several‑fold speedup.
This article provides a comprehensive overview of Hadoop, covering its origins, core components such as HDFS, MapReduce, and YARN, their architectures, data storage and processing mechanisms, fault‑tolerance features, scheduling strategies, and practical optimization techniques for large‑scale distributed computing.
The article describes the design, architecture, and operational details of a unified logging framework at Ctrip's payment center, covering log production via a Log4j2 extension, Kafka‑Camus collection, Hive/ORC storage, MapReduce parsing optimizations, and governance strategies for massive daily TB‑scale data.
The article explains that Spark’s in‑memory processing, thread‑based task model, selective shuffle sorting, and flexible RDD/DAG architecture give it a significant performance advantage over Hadoop MapReduce’s disk‑heavy, process‑based batch execution.
This article explains the concept and workflow of shuffle in Hadoop MapReduce and Spark, covering map‑side buffering, spill and merge, reduce‑side copy‑merge‑reduce, the reasons for sorting and file merging, and compares Hash‑Shuffle and Sort‑Shuffle implementations with performance considerations.
Data skew in Hive MapReduce jobs, caused by uneven key distribution during joins, group‑by, or COUNT(DISTINCT) operations, can severely slow tasks, and the article explains common scenarios and practical solutions such as using MapJoin, enabling map‑side aggregation, load‑balancing, and rewriting queries to mitigate skew.