batch processing

Collection size

120 articles

Page 1 of 6

JD Retail Technology

Sep 25, 2024 · Big Data

From a Personal Journey to Data Platform Architecture: Insights on Big Data, Cloud Computing, and System Design

The article narrates the author’s 30‑year programming career and shares technical reflections on building business‑agnostic, configurable data platforms, covering batch, streaming, interactive computing, big‑data sharding, Spark, Flink, cloud migration, and the philosophy of software architecture.

Big DataData Engineeringbatch processing

0 likes · 23 min read

From a Personal Journey to Data Platform Architecture: Insights on Big Data, Cloud Computing, and System Design

Mike Chen's Internet Architecture

Aug 16, 2024 · Big Data

Understanding the Lambda Architecture for Big Data Processing

This article explains the Lambda architecture—a three‑layer model combining batch and real‑time processing for large‑scale data, outlines its components, advantages, disadvantages, common tools, and compares it with the Kappa alternative while providing practical insights for data engineers.

Big DataData EngineeringLambda architecture

0 likes · 5 min read

Understanding the Lambda Architecture for Big Data Processing

DeWu Technology

Oct 28, 2024 · Backend Development

Design and Evolution of an E-commerce Batch Processing System

The article traces the evolution of an e‑commerce batch‑processing system—from an initial centralized workflow with reusable components, through a platform‑driven configuration and SPI registration, to a localized, asynchronous task‑reporting architecture employing priority queues and isolated thread pools—to balance flexibility, scalability, and operational risk.

ConfigurationJavabatch processing

0 likes · 18 min read

Design and Evolution of an E-commerce Batch Processing System

Wukong Talks Architecture

Apr 27, 2025 · Backend Development

Evolution of a Batch Processing System: From Centralized to Configurable and Localized Architectures

This article examines the evolution of a merchant batch processing system, detailing its transition from a centralized, tightly‑coupled architecture to a configurable, SPI‑based design and finally to a localized task‑reporting model, while discussing code reuse, scheduling strategies, isolation techniques, and performance challenges.

ConfigurationJavabackend development

0 likes · 18 min read

Evolution of a Batch Processing System: From Centralized to Configurable and Localized Architectures

Architecture Digest

Oct 11, 2024 · Backend Development

Common Interface Performance Optimization Techniques

This article outlines a series of backend interface performance optimization strategies—including batch processing, asynchronous execution, caching, preprocessing, pooling, parallelization, indexing, transaction management, program structure refactoring, deep pagination, SQL tuning, and lock granularity—to help developers reduce latency and improve system efficiency.

BackendIndexingOptimization

0 likes · 9 min read

Common Interface Performance Optimization Techniques

Architect

Apr 9, 2025 · Backend Development

Merging Requests and Batch Querying in Spring Boot to Reduce Database Connections

This article explains how to merge concurrent user requests into a single batch SQL query using Java's LinkedBlockingQueue, ScheduledThreadPoolExecutor and CompletableFuture in a Spring Boot application, thereby saving database connections and improving performance under high concurrency.

Database OptimizationJava ConcurrencySpring Boot

0 likes · 13 min read

Merging Requests and Batch Querying in Spring Boot to Reduce Database Connections

Top Architect

Jan 16, 2025 · Backend Development

Optimizing XML‑to‑MySQL Data Import: Reducing Execution Time from 300 s to 4 s

This article describes how to dramatically speed up the import of 60,000+ XML records into MySQL by analyzing the original environment, measuring baseline performance, and applying a series of backend optimizations—including MySQL batch mode, rewriteBatchedStatements, multithreaded asynchronous writes with Disruptor, and XML parsing improvements—ultimately cutting the total runtime from five minutes to just a few seconds.

DisruptorJavaMySQL

0 likes · 14 min read

Optimizing XML‑to‑MySQL Data Import: Reducing Execution Time from 300 s to 4 s

Architect's Tech Stack

Mar 17, 2025 · Backend Development

Comprehensive Interface Performance Optimization Strategies

This article presents a systematic guide to improving API response times by applying batch processing, asynchronous execution, caching, pooling, parallelism, indexing, transaction management, pagination, SQL tuning, and proper lock granularity, supplemented with practical Java code examples and diagrams.

SQL paginationTransaction Managementasynchronous

0 likes · 10 min read

Comprehensive Interface Performance Optimization Strategies

DataFunTalk

Apr 9, 2025 · Big Data

Highlights of the Apache Hudi Asia Technical Salon Hosted by Kuaishou – Practices and Innovations from Leading Companies

The Kuaishou‑hosted Apache Hudi Asia technical salon gathered over 230 attendees and featured seven experts from Kuaishou, Meituan, TikTok, Huawei, JD and others, who shared best practices, architecture designs, and performance optimizations for large‑scale data lake applications across AI, BI, and real‑time workloads.

AIApache HudiBig Data

0 likes · 14 min read

Highlights of the Apache Hudi Asia Technical Salon Hosted by Kuaishou – Practices and Innovations from Leading Companies

DataFunSummit

Nov 23, 2024 · Big Data

Bilibili's Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practice

This article presents Bilibili's end‑to‑end exploration of a streaming‑batch unified data pipeline built on Apache Iceberg, detailing the original and iterated architectures for massive user behavior transmission, online AI training, DB synchronization, and dimension‑join, along with performance gains, cost savings, and future plans.

FlinkIcebergOptimization

0 likes · 20 min read

Bilibili's Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practice

DataFunSummit

Jul 12, 2024 · Big Data

Data Lake Development Trends, Architecture, Integration, Lakehouse Core Capabilities, and Open Design

This article examines the current evolution of data lakes, detailing their overall architecture, batch and real‑time integration methods, Lakehouse core functionalities such as enhanced DML, schema evolution, ACID support, and open‑design principles that enable multi‑cloud deployment and seamless interaction with diverse compute engines.

LakehouseOpen Data FormatsReal-time Integration

0 likes · 12 min read

Data Lake Development Trends, Architecture, Integration, Lakehouse Core Capabilities, and Open Design

Rare Earth Juejin Tech Community

May 30, 2025 · Operations

Batch Convert Word Documents to PDF with Python and Win32COM

This guide demonstrates how to create a Python script that uses the win32com library to batch convert Word (.doc/.docx) files to PDF on Windows, including code examples, handling of Office/WPS detection, folder management, common pitfalls, and packaging the script into an executable.

AutomationPDFPython

0 likes · 10 min read

Batch Convert Word Documents to PDF with Python and Win32COM

Java Captain

Mar 21, 2025 · Backend Development

Request Merging and Batch Processing in Java Spring Boot to Reduce Database Connections

This article explains how to merge multiple user‑detail requests into a single database query using a blocking queue, scheduled thread pool, and CompletableFuture in Spring Boot, providing code examples, a high‑concurrency test, and discussion of trade‑offs such as added latency and timeout handling.

CompletableFutureJavaQueue

0 likes · 13 min read

Request Merging and Batch Processing in Java Spring Boot to Reduce Database Connections

Code Ape Tech Column

Jun 4, 2025 · Backend Development

Request Collapsing Techniques: Hystrix Collapser, BatchCollapser, and ConcurrentHashMultiset

This article compares three request‑collapsing techniques—Hystrix Collapser, a custom BatchCollapser, and Guava's ConcurrentHashMultiset—explaining their designs, configurations, code implementations, and suitable scenarios for reducing downstream load in Java backend systems.

BackendHystrixJava

0 likes · 13 min read

Request Collapsing Techniques: Hystrix Collapser, BatchCollapser, and ConcurrentHashMultiset

Python Programming Learning Circle

Dec 13, 2024 · Artificial Intelligence

Batch Image Translation Demo Using Youdao OCR API with Python

This article demonstrates how to build a Python desktop application that batch‑processes cosmetic product images, sends them to Youdao's OCR translation service, and displays the translated text, covering API preparation, request parameters, signature generation, and full source code.

Image TranslationOCRPython

0 likes · 12 min read

Batch Image Translation Demo Using Youdao OCR API with Python

Lobster Programming

Apr 3, 2025 · Databases

How to Quickly Generate Batch SQL Statements Using Excel

This tutorial explains how to import data into Excel, craft formulas that concatenate cell values into UPDATE or INSERT SQL statements, and drag the formulas down to produce batch SQL commands for efficient database operations.

AutomationDatabaseExcel

0 likes · 2 min read

How to Quickly Generate Batch SQL Statements Using Excel

Java Captain

Jun 10, 2025 · Backend Development

Why Spring Batch? Real‑World Scenarios, Core Architecture and Hands‑On Guide

This article explains the necessity of batch processing, presents typical use cases such as daily interest calculation, e‑commerce order archiving, log analysis and medical data migration, then dives deep into Spring Batch's core components, provides step‑by‑step code examples, performance‑tuning tips, production‑grade fault‑tolerance, monitoring solutions and a comprehensive FAQ.

JavaSpring Batchbatch processing

0 likes · 20 min read

Why Spring Batch? Real‑World Scenarios, Core Architecture and Hands‑On Guide

macrozheng

Mar 28, 2025 · Backend Development

Boost API Performance: 12 Proven Backend Optimization Techniques

This article presents a comprehensive set of backend optimization strategies—including batch processing, asynchronous execution, caching, pre‑processing, pooling, parallelism, indexing, transaction management, program refactoring, pagination, SQL tuning, and fine‑grained locking—to dramatically reduce API latency and improve system efficiency.

API optimizationSQL indexingasynchronous

0 likes · 10 min read

Boost API Performance: 12 Proven Backend Optimization Techniques

macrozheng

Mar 5, 2025 · Backend Development

How to Merge Concurrent Requests in Spring Boot to Save Database Connections

This article explains how to combine multiple simultaneous user requests on the server side using a queue, scheduled thread pool and CompletableFuture in Spring Boot, reducing database connections while handling high concurrency, and discusses implementation details, testing, and potential pitfalls.

JavaSpring Bootbatch processing

0 likes · 15 min read

How to Merge Concurrent Requests in Spring Boot to Save Database Connections

macrozheng

Feb 11, 2025 · Databases

Speed Up XML‑to‑MySQL Imports: Reduce 300 s to 4 s with JDBC Batch & Async

This article walks through optimizing a Java‑based XML‑to‑MySQL import, showing how to cut processing time from 300 seconds to just 4 seconds by enabling JDBC batch writes, using rewriteBatchedStatements, applying async writes with Disruptor, and tuning MySQL settings.

JDBCJavaMySQL

0 likes · 12 min read

Speed Up XML‑to‑MySQL Imports: Reduce 300 s to 4 s with JDBC Batch & Async