Tag

Pre-aggregation

1 views collected around this technical thread.

DataFunTalk
DataFunTalk
Jul 14, 2023 · Databases

Implementing Real‑Time Materialized Views to Accelerate Large‑Scale Time‑Series Queries

This article explains how to implement real‑time materialized views to accelerate large‑scale time‑series data queries, covering the need for materialized views, their definition, storage, incremental updates, pre‑computation, query partitioning, performance testing, and future directions.

Data PartitioningPre-aggregationQuery Acceleration
0 likes · 16 min read
Implementing Real‑Time Materialized Views to Accelerate Large‑Scale Time‑Series Queries
DataFunTalk
DataFunTalk
Dec 6, 2022 · Databases

Performance Optimization of Apache Doris for A/B Experiment Queries at Xiaomi

This article analyzes the performance bottlenecks of A/B experiment report queries on Apache Doris at Xiaomi, presents data-driven insights on query latency, field usage, and experiment ID matching, and details a series of optimizations—including pre‑aggregation, materialized views, bitmap deduplication, and schema redesign—that reduced query times by up to 60× and lowered cluster load.

A/B TestingApache DorisBitmap
0 likes · 17 min read
Performance Optimization of Apache Doris for A/B Experiment Queries at Xiaomi
Shopee Tech Team
Shopee Tech Team
Sep 23, 2021 · Big Data

Design and Architecture of the Boussole Real-Time Multi-Dimensional Data Analysis Engine

Boussole is Shopee’s real‑time analytics engine that transforms each dimension into key‑value pairs stored primarily in HBase, pre‑aggregates selected dimension combos, hashes metrics and tags, executes distributed PromQL queries with a CockroachDB‑inspired executor, applies Delta‑of‑Delta compression and point‑capping, and continues to evolve with adaptive pre‑aggregation and new storage models to maintain millisecond latency for massive multi‑dimensional analysis.

Big DataDistributed QueryPre-aggregation
0 likes · 24 min read
Design and Architecture of the Boussole Real-Time Multi-Dimensional Data Analysis Engine