Author

Big Data Technology Architecture

Exploring Open Source Big Data and AI Technologies

290

Articles

Likes

707

Views

Comments

Latest from Big Data Technology Architecture

100 recent articles max

Big Data Technology Architecture

Jun 2, 2021 · Big Data

Practical Operations of NetEase Big Data Platform: Architecture, EasyOps, Monitoring, and Experience Sharing

The presentation details NetEase's big data platform operations, covering current usage, the internally built EasyOps control system, a generic service‑operation framework based on Ansible, Prometheus‑Grafana monitoring, configuration management, network and storage optimizations, and lessons learned from cloud migration.

Big DataEasyOpsMonitoring

0 likes · 9 min read

Practical Operations of NetEase Big Data Platform: Architecture, EasyOps, Monitoring, and Experience Sharing

Big Data Technology Architecture

May 31, 2021 · Big Data

Practical Experience of Using Flink + Iceberg 0.11 on Qunar Data Platform

This article presents Qunar's practical experience with Flink and Iceberg 0.11, covering background challenges such as Kafka data loss and Hive metadata pressure, explaining Iceberg architecture, query planning, and detailed solutions including real‑time ingestion, small‑file handling, sorting, and code examples for seamless migration.

FlinkIcebergReal-time Processing

0 likes · 12 min read

Practical Experience of Using Flink + Iceberg 0.11 on Qunar Data Platform

Big Data Technology Architecture

May 19, 2021 · Databases

Combining HBase and Elasticsearch: Challenges and the Lindorm Searchindex Solution

This article examines the complementary strengths of HBase and Elasticsearch, outlines three integration patterns and their associated challenges, and introduces Alibaba Cloud's Lindorm Searchindex as a SQL‑driven, low‑cost solution that simplifies storage and full‑text search for massive data workloads.

ElasticsearchHBaseLindorm

0 likes · 12 min read

Combining HBase and Elasticsearch: Challenges and the Lindorm Searchindex Solution

Big Data Technology Architecture

May 13, 2021 · Big Data

Real-Time OLAP Evolution and Production Optimization at BTC.com

This article details BTC.com’s journey from a legacy batch‑oriented analytics stack to a modern real‑time OLAP architecture using Flink, ClickHouse, Kafka, and Kubernetes, highlighting the business drivers, technical choices, architectural evolution, optimizations, and future directions.

BlockchainClickHouseFlink

0 likes · 9 min read

Real-Time OLAP Evolution and Production Optimization at BTC.com

Big Data Technology Architecture

May 12, 2021 · Big Data

End-to-End Tutorial: Sync MySQL Binlog to Kafka and Consume with Flink Using TiDB

This article provides a step‑by‑step guide to build a data pipeline that captures MySQL binlog, streams it through Canal into Kafka, processes it with Flink, and finally writes the results into TiDB, covering environment setup, component deployment, configuration, and verification.

CanalFlinkTiDB

0 likes · 31 min read

End-to-End Tutorial: Sync MySQL Binlog to Kafka and Consume with Flink Using TiDB

Big Data Technology Architecture

May 6, 2021 · Databases

Elasticsearch Pagination: From+size, search_after, and Scroll – Differences, Advantages, and Use Cases

This article explains Elasticsearch’s three pagination methods—From + size, search_after, and Scroll—detailing their definitions, code examples, advantages, disadvantages, and suitable scenarios, while also discussing max_result_window limits, PIT views, and best practices for handling large result sets.

ElasticsearchSearchbackend

0 likes · 13 min read

Elasticsearch Pagination: From+size, search_after, and Scroll – Differences, Advantages, and Use Cases

Big Data Technology Architecture

May 6, 2021 · Big Data

Using Spark SQL to Operate on Apache Hudi Tables – Step‑by‑Step Guide

This tutorial demonstrates how to use Spark SQL to create, insert, update, delete, merge, and drop Apache Hudi tables, covering environment setup, Spark‑SQL launch, configuration, and a series of SQL commands with example outputs.

Apache HudiSQLSpark SQL

0 likes · 7 min read

Using Spark SQL to Operate on Apache Hudi Tables – Step‑by‑Step Guide

Big Data Technology Architecture

Apr 25, 2021 · Databases

ClickHouse vs Elasticsearch: Choosing a Database for Log Storage and Analysis

The article compares ClickHouse, Elasticsearch, and MySQL for log storage and analysis, highlighting ClickHouse's active development, SQL support, JSON handling, performance, storage efficiency, integration options, and practical query examples to help developers decide the best backend for observability pipelines.

AnalyticsJSONlog storage

0 likes · 12 min read

ClickHouse vs Elasticsearch: Choosing a Database for Log Storage and Analysis

Big Data Technology Architecture

Apr 19, 2021 · Big Data

Reframing Apache Hudi as a Data Lake Platform: Vision, Capabilities, and Future Directions

Apache Hudi is being re‑positioned from a simple table format to a full‑featured data lake platform, offering transactional storage, MVCC concurrency, metadata services, Deltastreamer ingestion, and plans for cache and timeline metadata services, aligning its vision with modern lakehouse architectures.

Apache HudiTransactional Storagemetadata

0 likes · 5 min read

Reframing Apache Hudi as a Data Lake Platform: Vision, Capabilities, and Future Directions

Big Data Technology Architecture

Apr 15, 2021 · Databases

HBase Read Performance Optimization: Best Practices and Tuning Guide

This article presents a comprehensive, practical guide to diagnosing and optimizing HBase read latency, covering common issues such as Full GC, region‑in‑transition, low write throughput, and high read delay, and offering client‑side, server‑side, column‑family, and HDFS tuning recommendations.

HBase

0 likes · 15 min read