Adaptive Query Execution

DataFunSummit

Sep 27, 2022 · Big Data

Apache Spark Adaptive Query Execution and Kyuubi Optimization Practices for Data Warehousing

This article presents a detailed overview of Apache Spark's Adaptive Query Execution evolution, its optimization techniques, and performance gains, followed by an in‑depth discussion of Apache Kyuubi's architecture, security integrations, cloud‑native capabilities, and practical Rebalance + Z‑Order strategies that enhance data‑warehouse task efficiency and query performance.

Adaptive Query ExecutionApache SparkBig Data Optimization

0 likes · 19 min read

Apache Spark Adaptive Query Execution and Kyuubi Optimization Practices for Data Warehousing

Big Data Technology Architecture

Nov 16, 2021 · Big Data

Understanding Adaptive Query Execution and Dynamic Partition Pruning in Apache Spark 3.0

This article explains how Apache Spark 3.0 improves SQL workload performance through Adaptive Query Execution (AQE) and Dynamic Partition Pruning (DPP), detailing their design principles, runtime optimizations, configuration parameters, and practical examples that demonstrate reduced shuffle partitions, smarter join strategies, and handling of data skew.

Adaptive Query ExecutionBig DataDynamic Partition Pruning

0 likes · 9 min read

Understanding Adaptive Query Execution and Dynamic Partition Pruning in Apache Spark 3.0

Big Data Technology Architecture

Aug 12, 2020 · Big Data

Overview of New Features and Improvements in Apache Spark 3.0

Apache Spark 3.0 introduces a suite of performance enhancements, richer APIs, improved monitoring, SQL compatibility, new data sources, and ecosystem extensions, including Adaptive Query Execution, Dynamic Partition Pruning, Join Hints, pandas UDF improvements, and accelerator‑aware scheduling, to boost scalability and ease of use for big‑data workloads.

Adaptive Query ExecutionApache SparkBig Data

0 likes · 15 min read

Overview of New Features and Improvements in Apache Spark 3.0

Big Data Technology Architecture

Aug 8, 2020 · Big Data

Overview of SQL Performance Improvements in Apache Spark 3.0

Apache Spark 3.0 introduces extensive SQL performance enhancements, including a new explain format, expanded join hints, adaptive query execution, dynamic partition pruning, enhanced nested column pruning, improved aggregation code generation, and support for newer Scala and Java versions, all aimed at optimizing query planning and execution.

Adaptive Query ExecutionApache SparkBig Data

0 likes · 14 min read

Overview of SQL Performance Improvements in Apache Spark 3.0

Big Data Technology Architecture

Jun 20, 2020 · Big Data

Apache Spark 3.0.0 Release: New Features, Improvements, and Timeline

Apache Spark 3.0.0, released after a 21‑month development cycle and several preview and release‑candidate votes, introduces major enhancements such as Dynamic Partition Pruning, Adaptive Query Execution, accelerator‑aware scheduling, DataSource V2, expanded pandas UDFs, new join hints, richer monitoring, SparkR vectorization, Kafka header support, and broader ecosystem integrations, while fixing over 3,400 issues.

Adaptive Query ExecutionApache SparkBig Data

0 likes · 17 min read

Apache Spark 3.0.0 Release: New Features, Improvements, and Timeline