BestHub
Sign in
Tag

Vectorization

1 views collected around this technical thread.

Didi Tech
Didi Tech
Mar 27, 2025 · Operations

Performance Optimization and Architecture of iLogTail for High‑Scale Log Collection

Didi replaced its legacy agent with Alibaba’s open‑source iLogTail, re‑architected it to use a shared thread‑pool and SIMD‑accelerated parsing, rewrote critical plugins in C++ and added robust Kafka retry logic, achieving over twice the throughput while cutting CPU usage by more than half and maintaining near‑zero latency at massive scale.

C++KafkaPerformance Optimization
0 likes · 10 min read
Performance Optimization and Architecture of iLogTail for High‑Scale Log Collection
DataFunSummit
DataFunSummit
Feb 22, 2025 · Big Data

Blaze Engine: A Rust‑Based Native Vectorized Execution Engine for Spark SQL

The article introduces Blaze, Kuaishou's Rust‑powered native execution engine that vectorizes Spark SQL workloads, explains its architecture and operation, presents benchmark results showing up to 50% latency reduction, and details internal deployments, industry case studies, community collaborations, and the 2025 roadmap.

Big DataNative ExecutionPerformance Optimization
0 likes · 12 min read
Blaze Engine: A Rust‑Based Native Vectorized Execution Engine for Spark SQL
DataFunSummit
DataFunSummit
Feb 1, 2025 · Big Data

Spark Native and Cloud Native: Vectorized SQL Engines, Remote Shuffle, and EMR Serverless Spark Practices

This article explains the challenges of big‑data processing in the cloud era, introduces Spark’s native‑language SQL engine rewrites, discusses vectorization and code generation techniques, describes cloud‑native storage‑compute separation with Remote Shuffle services such as Apache Celeborn, and presents the production benefits of Alibaba Cloud’s EMR Serverless Spark.

Big DataCodegenEMR Serverless
0 likes · 12 min read
Spark Native and Cloud Native: Vectorized SQL Engines, Remote Shuffle, and EMR Serverless Spark Practices
AntData
AntData
Dec 11, 2024 · Big Data

Flex: A Stream‑Batch Integrated Vectorized Engine for Flink

This article introduces Flex, a Flink‑compatible stream‑batch vectorized engine built on Velox and Gluten, explains the SIMD‑based execution model, details native operator optimizations, fallback mechanisms, correctness and usability improvements, and presents performance results and future development plans.

FlinkPerformanceSIMD
0 likes · 17 min read
Flex: A Stream‑Batch Integrated Vectorized Engine for Flink
DataFunSummit
DataFunSummit
Aug 17, 2024 · Big Data

AnalyticDB Spark Architecture and Vectorized Engine Performance Overview

This article introduces the AnalyticDB Spark architecture, explains the need for Spark vectorization, surveys industry vectorized solutions, details ADB Spark's own vectorized implementation with Gluten and Velox, and presents performance test results showing a 6.98‑fold speedup over open‑source Spark.

AnalyticDBBig DataGluten
0 likes · 9 min read
AnalyticDB Spark Architecture and Vectorized Engine Performance Overview
Tencent Cloud Developer
Tencent Cloud Developer
Jul 11, 2024 · Databases

LibraDB Execution Engine Architecture Evolution and Optimization

LibraDB, the column‑store replica of TDSQL MySQL, has evolved its execution engine from a simple scatter‑gather model to a vectorized SMP pipeline that integrates MPP parallelism, asynchronous I/O, SIMD‑accelerated aggregation and join operators, work‑stealing, and runtime filters, thereby fully exploiting CPU, memory, network and disk resources for both OLTP and analytical queries.

DatabaseExecution EngineHash Join
0 likes · 22 min read
LibraDB Execution Engine Architecture Evolution and Optimization
Python Programming Learning Circle
Python Programming Learning Circle
Jan 4, 2024 · Fundamentals

Simple Methods to Speed Up Python For Loops (1.3× to 970×)

This article presents a series of practical techniques—such as list comprehensions, pre‑computing lengths, using sets, skipping irrelevant iterations, inlining functions, generators, map, memoization, vectorization, and efficient string joining—that can accelerate Python for‑loops anywhere from 1.3‑fold up to 970‑fold, with concrete benchmark results and code examples.

PerformancePythonVectorization
0 likes · 15 min read
Simple Methods to Speed Up Python For Loops (1.3× to 970×)
DataFunSummit
DataFunSummit
Dec 16, 2023 · Databases

Optimizing Precise Deduplication with Doris Bitmap: Architecture, Performance Enhancements, and Practical Practices

This article presents a comprehensive overview of precise deduplication in Meituan's Doris database, detailing the underlying bitmap data structures, aggregation bottlenecks, and a series of optimizations—including memory management, fast union, orthogonal encoding, and vectorized engine integration—that together achieve significant performance gains in high‑cardinality scenarios.

BitMapDatabaseDeduplication
0 likes · 20 min read
Optimizing Precise Deduplication with Doris Bitmap: Architecture, Performance Enhancements, and Practical Practices
DataFunSummit
DataFunSummit
May 21, 2023 · Big Data

Blaze: Design and Practice of SparkSQL Native Operator Optimization at Kuaishou

This article presents Blaze, a Kuaishou‑built native execution middleware for SparkSQL that leverages Apache DataFusion to achieve vectorized operator execution, detailing its architecture, implementation, performance gains, current coverage, benchmark results, production rollout, and future development plans.

Big DataDataFusionNative Execution
0 likes · 17 min read
Blaze: Design and Practice of SparkSQL Native Operator Optimization at Kuaishou
Python Programming Learning Circle
Python Programming Learning Circle
Mar 31, 2023 · Fundamentals

Vectorized String Operations in Pandas: Methods and Examples

This article explains how Pandas' vectorized string operations enable efficient, loop‑free processing of text data, covering basic methods like len() and lower(), advanced regex functions, and additional utilities such as split, replace, slice, and get_dummies, with code examples and usage details.

Vectorizationdata cleaningpandas
0 likes · 21 min read
Vectorized String Operations in Pandas: Methods and Examples
DataFunSummit
DataFunSummit
Mar 29, 2023 · Big Data

Gluten Vectorized Engine: Boosting Spark Performance with Native Execution

The article introduces the Gluten vectorized engine, explains why Spark’s CPU bottleneck motivates integrating native vectorized back‑ends via Substrait, details its architecture, component design, current performance gains of up to three‑fold, and outlines ongoing development and future work.

Big DataGlutenNative Engine
0 likes · 18 min read
Gluten Vectorized Engine: Boosting Spark Performance with Native Execution
DataFunTalk
DataFunTalk
Nov 14, 2022 · Databases

Performance Optimization and Tuning of Apache Doris Vectorized Version for Xiaomi's A/B Experiment Platform

Xiaomi upgraded its Apache Doris from version 0.13 to the vectorized 1.1.2 release for its A/B experiment platform, conducting extensive single‑SQL and concurrent tests, identifying CPU, memory, and fragment timeout issues, and applying tuning such as memory decommit settings, string matching improvements, and patches to achieve up to 5× query speed gains and enhanced stability.

A/B testingApache DorisDatabase Optimization
0 likes · 20 min read
Performance Optimization and Tuning of Apache Doris Vectorized Version for Xiaomi's A/B Experiment Platform
DataFunSummit
DataFunSummit
Oct 27, 2022 · Databases

Vectorized Storage Layer Refactoring in Apache Doris: Design, Implementation, and Performance Evaluation

This article explains the motivation, design, and implementation of vectorizing Apache Doris's storage layer using SIMD techniques, covering engine overview, vectorized programming concepts, storage architecture, index and predicate optimizations, delayed materialization, output improvements, and performance test results.

Apache DorisOLAPPerformance Optimization
0 likes · 13 min read
Vectorized Storage Layer Refactoring in Apache Doris: Design, Implementation, and Performance Evaluation
Model Perspective
Model Perspective
Oct 10, 2022 · Fundamentals

Matrix-to-Matrix Derivatives: Definitions, Differential Method & Examples

This article explains the definition of matrix‑to‑matrix derivatives, introduces the vectorization‑based differential approach using Kronecker products, presents key matrix‑vectorization properties, and walks through detailed examples illustrating how to compute such derivatives, highlighting their role and limitations in machine‑learning optimization.

Kronecker productVectorizationderivative
0 likes · 5 min read
Matrix-to-Matrix Derivatives: Definitions, Differential Method & Examples
Model Perspective
Model Perspective
Jun 28, 2022 · Fundamentals

Master NumPy: Visual Guide to Multidimensional Arrays and Operations

An in‑depth visual tutorial explains NumPy’s core concepts—from one‑dimensional vectors to high‑dimensional tensors—covering array creation, indexing, arithmetic, broadcasting, sorting, and advanced functions like meshgrid and einsum, empowering developers to harness efficient multidimensional computations in Python.

Multidimensional ArraysNumPyPython
0 likes · 21 min read
Master NumPy: Visual Guide to Multidimensional Arrays and Operations
DataFunSummit
DataFunSummit
Mar 21, 2022 · Databases

Vectorization in Apache Doris: Design, Implementation, and Future Roadmap

This article explains how Apache Doris adopts CPU‑level vectorization and columnar storage to boost query performance, details the design and current status of its vectorized engine, and outlines future work such as JOIN acceleration, storage‑layer vectorization, import optimization, and extensive SQL function support.

Apache DorisPerformance OptimizationSIMD
0 likes · 21 min read
Vectorization in Apache Doris: Design, Implementation, and Future Roadmap
DataFunTalk
DataFunTalk
Feb 27, 2022 · Databases

Vectorization in Apache Doris: Design, Implementation, Current Status, and Future Plans

This article explains how Apache Doris adopts CPU vectorization techniques—such as SIMD, columnar storage, and cache‑friendly designs—to boost query performance, detailing its current vectorized engine architecture, recent benchmarks, ongoing work on JOIN, storage, import, and future enhancements.

Apache DorisDatabase PerformanceSIMD
0 likes · 22 min read
Vectorization in Apache Doris: Design, Implementation, Current Status, and Future Plans