Batch Processing

Collection size

121 articles

Page 4 of 7

macrozheng

Mar 28, 2025 · Backend Development

Boost API Performance: 12 Proven Backend Optimization Techniques

This article presents a comprehensive set of backend optimization strategies—including batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelism, indexing, transaction management, program refactoring, pagination, SQL tuning, and fine‑grained locking—to dramatically reduce API latency and improve system efficiency.

API optimizationBatch ProcessingSQL indexing

0 likes · 10 min read

Boost API Performance: 12 Proven Backend Optimization Techniques

macrozheng

Mar 5, 2025 · Backend Development

How to Merge Concurrent Requests in Spring Boot to Save Database Connections

This article explains how to combine multiple simultaneous user requests on the server side using a queue, scheduled thread pool and CompletableFuture in Spring Boot, reducing database connections while handling high concurrency, and discusses implementation details, testing, and potential pitfalls.

Batch ProcessingJavaSpring Boot

0 likes · 15 min read

How to Merge Concurrent Requests in Spring Boot to Save Database Connections

macrozheng

Feb 11, 2025 · Databases

Speed Up XML‑to‑MySQL Imports: Reduce 300 s to 4 s with JDBC Batch & Async

This article walks through optimizing a Java‑based XML‑to‑MySQL import, showing how to cut processing time from 300 seconds to just 4 seconds by enabling JDBC batch writes, using rewriteBatchedStatements, applying async writes with Disruptor, and tuning MySQL settings.

Batch ProcessingJDBCJava

0 likes · 12 min read

Speed Up XML‑to‑MySQL Imports: Reduce 300 s to 4 s with JDBC Batch & Async

macrozheng

Jul 28, 2021 · Backend Development

Master Spring Batch: Core Concepts, Architecture, and Best Practices

This article provides a comprehensive overview of Spring Batch, covering its purpose, architecture, core components such as Job, Step, ItemReader/Writer/Processor, execution contexts, chunk processing, skip strategies, and practical tips for configuration and memory management.

Backend DevelopmentBatch ProcessingChunk Processing

0 likes · 20 min read

Master Spring Batch: Core Concepts, Architecture, and Best Practices

DeWu Technology

Oct 10, 2022 · Big Data

Offline and Real-Time User Profile Fusion Architecture

The architecture combines a nightly batch job that generates offline user profiles stored in HBase with a Flink‑based stream layer that lazily loads those profiles on app start and creates real‑time updates, then fuses both streams into a unified, timestamp‑ordered profile in Redis, forming a Lambda‑style pipeline.

Batch ProcessingFlinkHBase

0 likes · 10 min read

DaTaobao Tech

Aug 11, 2022 · Big Data

Unify SQL Engine: Integrating Stream, Batch, and Online Computing for Data Warehousing

The article describes how fragmented real‑time, batch, and online data‑warehouse pipelines suffer from low productivity and inconsistent data quality, and introduces a unified SQL engine built on Apache Calcite that parses, optimizes, and compiles a single SQL statement into executable plans for ODPS, Flink, or Java, leveraging Janino code generation, multi‑backend state storage, and snapshot‑join semantics to boost performance and simplify development.

Batch ProcessingCalciteFlink

0 likes · 16 min read

Unify SQL Engine: Integrating Stream, Batch, and Online Computing for Data Warehousing

Tencent Cloud Developer

Aug 1, 2019 · Databases

FeatureKV: A High-Performance Key-Value Storage System for WeChat's Billion-Scale Challenges

FeatureKV, WeChat’s high‑performance key‑value store, handles one‑billion queries per second and ingests a billion keys per hour by separating write‑only DataSvr from read‑only KVSvr, supporting in‑memory, indexed, and block‑indexed tables, scaling horizontally, guaranteeing eventual consistency with versioned reads, and delivering up to 11 billion reads per second with sub‑15 ms latency.

Batch ProcessingDistributed StorageFeatureKV

0 likes · 22 min read

FeatureKV: A High-Performance Key-Value Storage System for WeChat's Billion-Scale Challenges

JD Retail Technology

Feb 29, 2024 · Databases

Optimizing Large‑Scale Batch Processing for an Advertising Platform: From Query Tuning to Load‑Balanced Execution

This article presents a real‑world case study of optimizing massive batch‑processing tasks in an ad‑platform by applying query‑level improvements, cursor‑based pagination, shard‑aware batch updates, JVM‑tuned garbage collection, and distributed load‑balancing, ultimately reducing CPU usage from 80% to under 2% and cutting query‑per‑minute volume from millions to a few thousand.

Batch ProcessingDatabase OptimizationJava

0 likes · 22 min read

Optimizing Large‑Scale Batch Processing for an Advertising Platform: From Query Tuning to Load‑Balanced Execution

Beike Product & Technology

Mar 10, 2020 · Fundamentals

Optimizing String Replacement Using SSE2 SIMD Instructions

This article explains how to use SSE2 SIMD instructions to optimize string replacement operations, demonstrating a 16-character batch processing technique that significantly improves performance for longer strings.

Batch ProcessingPerformanceSIMD

0 likes · 4 min read

Optimizing String Replacement Using SSE2 SIMD Instructions

Kuaishou Tech

Apr 9, 2021 · Backend Development

Design and Implementation of Red Packet Calculation and Distribution System for Spring Festival Activity

This article details the design of a red packet calculation and distribution system for a Spring Festival promotion, addressing mixed-type packet computation, seamless transition to awarding, distributed task processing, stability guarantees, and successful implementation results.

Batch ProcessingReal-time ProcessingRed Packet

0 likes · 12 min read

Design and Implementation of Red Packet Calculation and Distribution System for Spring Festival Activity

JD Tech

May 31, 2018 · Backend Development

Design and Architecture of a Unified MySQL Data Synchronization Platform

This article details the design of a unified MySQL data synchronization platform that consolidates offline sync, real‑time subscription, and real‑time sync into BatchJob, StreamJob, and PieJob abstractions, describing task implementations, cluster architecture, high‑availability mechanisms, and evolution challenges such as file loss and metadata handling.

Batch ProcessingHigh AvailabilityStream Processing

0 likes · 10 min read

Design and Architecture of a Unified MySQL Data Synchronization Platform

Ctrip Technology

Jan 25, 2017 · Backend Development

Handling Duplicate Messages, Ordering, Concurrency, and Batch Processing in Message‑Driven Systems

This article shares practical patterns and built‑in mechanisms for dealing with duplicate messages, message ordering, concurrent updates, asynchronous acknowledgments, and batch processing in a large‑scale, message‑driven architecture, illustrated with QMQ examples from Qunar's platform.

Backend DevelopmentBatch ProcessingDuplicate Message Handling

0 likes · 16 min read

Handling Duplicate Messages, Ordering, Concurrency, and Batch Processing in Message‑Driven Systems

Zhuanzhuan Tech

Jan 11, 2019 · Databases

Differences Between TiDB and MySQL: Transactions, Queries, Server‑Side Prepared Statements, and Batch Processing

This article examines TiDB, a world‑class open‑source distributed NewSQL database, comparing its transaction and query behavior with MySQL, discussing underlying Percolator model, server‑side prepared statements, batch processing techniques, and practical optimization strategies for developers.

Batch ProcessingDistributed DatabaseQuery Optimization

0 likes · 10 min read

Differences Between TiDB and MySQL: Transactions, Queries, Server‑Side Prepared Statements, and Batch Processing

Rare Earth Juejin Tech Community

Feb 24, 2024 · Backend Development

Introducing Karta: A Lightweight Go Library for Asynchronous and Batch Function Task Processing

This article introduces Karta, a lightweight Go library that provides two modes—Pipeline for unknown‑size asynchronous tasks and Group for known‑size batch tasks—offering a concise API, configurable workers, and built‑in callbacks to simplify high‑performance concurrent processing in backend applications.

Batch ProcessingGoasync

0 likes · 9 min read

Introducing Karta: A Lightweight Go Library for Asynchronous and Batch Function Task Processing

Big Data Technology Architecture

Nov 15, 2021 · Big Data

Flink Sort‑Shuffle: Design, Implementation, and Performance Evaluation

This article explains how Flink's new sort‑shuffle mechanism improves large‑scale batch processing by reducing file counts, optimizing I/O, lowering memory usage, and delivering up to tenfold speedups, while also detailing configuration tips and future enhancements.

Batch ProcessingData ShuffleFlink

0 likes · 16 min read

Flink Sort‑Shuffle: Design, Implementation, and Performance Evaluation

IT Architects Alliance

Apr 23, 2023 · Backend Development

Common Interface Performance Optimization Strategies

This article outlines a comprehensive set of backend interface optimization techniques—including batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelization, indexing, transaction sizing, code restructuring, deep pagination, SQL tuning, and lock granularity—to reduce latency and improve overall system efficiency.

API optimizationAsynchronous ExecutionBatch Processing

0 likes · 8 min read

Common Interface Performance Optimization Strategies

Java Architect Essentials

Dec 5, 2023 · Backend Development

Comprehensive Interface Performance Optimization Strategies

This article presents a collection of practical backend interface optimization techniques—including batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelism, indexing, transaction management, code restructuring, pagination, SQL tuning, and lock granularity—to reduce latency and improve overall system efficiency.

API optimizationBatch ProcessingPerformance

0 likes · 10 min read

Comprehensive Interface Performance Optimization Strategies

Architecture Digest

Oct 24, 2023 · Backend Development

Request Merging and Batch Processing in Spring Boot to Reduce Database Connections

This article explains how to merge multiple user requests on the server side, batch them into a single SQL query using a blocking queue and CompletableFuture, and handle high‑concurrency scenarios with scheduled tasks and timeout‑aware queues to save database connection resources.

Batch ProcessingJavaSpring Boot

0 likes · 13 min read

Request Merging and Batch Processing in Spring Boot to Reduce Database Connections

Architecture Digest

Mar 11, 2020 · Big Data

Apache Flink: Unified Stream and Batch Processing Architecture and Core Concepts

This article provides a comprehensive overview of Apache Flink, explaining how it unifies stream and batch processing on a single runtime, detailing its key features, APIs, libraries, architectural components, fault‑tolerance mechanisms, scheduling, iterative processing, and back‑pressure monitoring.

Apache FlinkBackpressureBatch Processing

0 likes · 20 min read

Apache Flink: Unified Stream and Batch Processing Architecture and Core Concepts

Code Ape Tech Column

Sep 16, 2023 · Backend Development

Batch Request Merging in Java to Reduce Database Connections

This article explains how to merge multiple user‑detail requests on the server side using a blocking queue, scheduled thread pool and CompletableFuture in Spring Boot, thereby converting many individual SQL calls into a single batch query, saving database connections and improving high‑concurrency performance.

Batch ProcessingCompletableFutureDatabase Optimization

0 likes · 13 min read

Batch Request Merging in Java to Reduce Database Connections