This article objectively compares NewSQL databases with middleware‑based sharding, dissecting their core architectures, distributed transaction handling, high‑availability designs, scaling mechanisms, SQL support, storage engines, and maturity to help engineers decide the most suitable solution for their workloads.
An in‑depth comparison of NewSQL databases and middleware‑based sharding reveals each approach’s architectural strengths, distributed transaction handling, scalability, HA mechanisms, storage engine design, and ecosystem maturity, guiding readers on when to adopt NewSQL versus traditional sharding solutions.
This article explains why MongoDB sharding is required for scaling storage and performance, describes the shard, config server, and mongos components, outlines range, hash, and compound sharding strategies, and provides practical guidance on shard key selection, balancing, backup, tuning, and security.
This article explains how to migrate sharded Mycat data to ActionDB with ActionOMS, addressing duplicate primary keys by modifying both source and target schemas, configuring migration tools, and completing validation and post‑migration steps.
A cross‑border e‑commerce company replaced its massive MySQL instance with OceanBase, detailing the business drivers, selection criteria, two‑phase migration process, performance gains, cost savings, remaining incompatibilities, and practical operational lessons learned.
This article examines the core challenges of distributed data systems—including consistency, availability, and partition tolerance—then details master‑slave replication mechanisms for MySQL and Redis, various replication modes and binlog formats, and explores data partitioning, sharding, and hot‑key mitigation techniques for scalable, high‑availability deployments.
This article explores the challenges of handling massive simultaneous requests in distributed architectures and presents practical solutions such as load balancing, distributed caching, asynchronous processing, and sharding, illustrated with case studies from major e‑commerce and social platforms.
This article examines seven typical challenges that arise after implementing database sharding—such as global ID conflicts, cross‑shard queries, distributed transactions, shard‑key design, data migration, pagination, and operational overhead—and provides practical solutions and code examples to address each issue.
This article analyzes the architectural impact of cross‑city multi‑active deployments, focusing on data‑layer design, write latency, sharding strategies, replication topologies, and routing considerations to achieve high availability, performance, and scalability in large‑scale distributed systems.
A ClickHouse cluster showed wildly varying row counts when querying sharded tables, while local tables behaved correctly; the article analyses the root cause in the cluster and table configuration, explains why the inconsistency occurs, and provides a step‑by‑step fix by switching to replicated tables.
When a MySQL table grows to tens of millions of rows, insert and query latency spikes, schema changes become painful, and only a subset of data is needed; this guide explains how to assess table size, why large tables slow down, and three practical solutions—partitioning, sharding, and hot/cold archiving—along with their trade‑offs and implementation steps.
This article explains common data‑migration strategies such as hash‑modulo, time‑range and ID‑range sharding, and compares practical migration solutions including downtime migration, dual‑write, and an MQ + Redis approach, with detailed workflow diagrams and implementation tips.
The article shares a senior architect’s experience with large‑scale MySQL tables, discussing why sharding and Elasticsearch were introduced, the pagination bug caused by Sharding‑JDBC, and a comprehensive set of hardware, software, and SQL tuning recommendations, including a full MySQL configuration example.
This article explains how the Go library BigCache uses sharding, efficient hash functions, and memory‑optimised data structures to deliver high‑concurrency, low‑latency caching, while avoiding GC bottlenecks and providing practical code examples and performance benchmarks.
This article describes how a senior architect designed, configured, and implemented a 50‑table sharding solution for loan and repayment request data using ShardingSphere, Spring Boot 3, MySQL, and custom synchronization scripts, while also addressing historical data migration, backend query changes, and operational safeguards.
This article examines the fundamental drawbacks of MySQL sharding in large‑scale e‑commerce, highlighting distributed transaction challenges, limited cross‑shard queries, global key constraints, scaling difficulties, operational overhead, and why distributed databases may be a better long‑term solution.
This article defines high concurrency, outlines QPS thresholds for small, medium, high, and ultra‑high traffic, and presents practical solutions such as multi‑level caching, load balancing, database sharding, and message‑queue traffic shaping to build robust backend systems.
This article explains why MySQL auto‑increment IDs can run out in large or special‑case tables and presents six practical solutions—including changing the column type to BIGINT, using UUIDs, segmenting IDs, composite keys, adjusting auto‑increment steps, and sharding databases—to ensure scalable and unique primary keys.
This article analyzes how write latency, write volume, isolation, and data replication strategies influence the design of multi‑active distributed systems, offering practical guidance on sharding, synchronous and asynchronous replication, routing, and architecture selection for high availability and performance across regions.
MongoDB 8.0 launches with up to 36% higher read throughput, 56% faster batch writes, 200% faster time‑series processing, 50‑fold faster sharding distribution, quantized vector search, and queryable encryption, delivering significant performance, cost, and security improvements for modern workloads.
This article walks through evaluating massive MySQL tables, explains why large row counts slow queries, and compares three practical solutions—table partitioning, database sharding, and hot/cold archiving—while highlighting their trade‑offs and offering guidance on selecting the right approach for a given workload.
This article objectively compares NewSQL distributed databases with traditional middleware‑based sharding, examining their architectural advantages, distributed transaction handling, high‑availability mechanisms, scaling and sharding strategies, storage engine differences, ecosystem maturity, and provides guidance on selecting the appropriate solution for various workloads.
This article explains how to assess MySQL table size using table capacity, disk usage, and instance resources, describes why large tables degrade performance, and presents three practical solutions—partitioning, sharding (horizontal/vertical), and hot‑cold data archiving—along with their trade‑offs and implementation tips.
This article introduces MyBatis‑Mate, an enterprise‑grade MyBatis extension that provides sharding, data auditing, sensitive‑word filtering, field encryption, dictionary binding, dynamic DDL maintenance, data‑scope permissions and performance monitoring, complete with practical code examples and configuration guidance.
This technical article describes how a senior architect designed, configured, and migrated loan and repayment application tables to a sharded MySQL setup using ShardingSphere, Spring Boot, and MyBatis‑Plus, covering schema design, historical data sync, dual‑write switches, code integration, and common pitfalls.
MongoDB reported a 13% revenue rise to $478.1 million in Q2 2025, highlighted a major win over PostgreSQL at Fanatics Betting & Gaming, and emphasized its flexible JSON‑like architecture, horizontal scaling, and sharding advantages while noting ongoing competition and a revised growth outlook.
The article introduces mybatis-mate, an enterprise‑grade MyBatis module that supports sharding, multi‑datasource, data auditing, field encryption, sensitive‑data masking, automatic DDL maintenance, performance logging, and fine‑grained data‑scope control, providing code examples and configuration guidelines for Spring Boot projects.
This article objectively compares NewSQL distributed databases with traditional middleware‑based sharding solutions, examining their architectural differences, transaction support, scalability, HA, storage engines, and ecosystem maturity, to help readers decide which approach best fits their performance, consistency, and operational needs.
This article objectively compares NewSQL databases with middleware‑based sharding solutions, examining architecture, distributed transactions, CAP constraints, high availability, scaling, SQL support, storage engines, and maturity to help readers choose the right approach for their workloads.
This article details the design, planning, and step‑by‑step implementation of a database table‑partitioning solution for loan and repayment request tables, covering schema decisions, sharding strategy, historical data migration, dynamic synchronization, code examples with ShardingSphere, SpringBoot, and operational considerations for seamless migration.
This article outlines the progressive evolution of database architectures—from simple monolithic designs to cold‑hot separation, read‑write splitting, and horizontal sharding—explaining each pattern, its components, and the performance and scalability benefits they bring to large‑scale systems.
This article outlines nine practical methods—caching, parallelization, batch processing, data compression, lock‑free design, sharding, request avoidance, pooling, and asynchronous handling—demonstrating how each can be applied to backend services to dramatically reduce latency and improve throughput.
After a year of business coding, the author details the design, implementation, and migration strategy for splitting loan and repayment tables into 50 sharded tables using ShardingSphere with Spring Boot, covering schema design, data synchronization, dynamic switches, and practical pitfalls.
This article explains the core concepts of distributed systems—including the CAP theorem, BASE theory, message‑queue challenges, Redis sentinel issues, sharding strategies, unique ID generation, and distributed transaction patterns—while offering practical guidance to prevent common pitfalls and improve reliability.
This article provides an in‑depth overview of MySQL fundamentals, covering join types, normalization, data types, index structures, locking mechanisms, transaction isolation, MVCC, replication, sharding, and practical performance‑tuning techniques for large‑scale databases.
This article provides a detailed tutorial on database sharding (分库分表), covering concepts, common sharding strategies, a step‑by‑step MyCAT configuration, Java JDBC implementation, and key considerations such as data distribution, complexity, transaction handling, and development cost.
This article objectively compares NewSQL distributed databases with traditional middleware‑based sharding solutions, examining their architectures, transaction models, scalability, high‑availability mechanisms, storage engines, maturity, and offering a decision framework to help engineers choose the most suitable approach for their workloads.
The article outlines practical backend interface performance optimizations—including proper indexing, SQL tuning, parallel remote calls, batch queries, asynchronous processing, scoped transactions, fine-grained locking, pagination batching, multi-level caching, sharding, and monitoring tools—to dramatically reduce latency and improve throughput.
This article introduces MyBatis-Mate, an official MyBatis‑Plus extension offering enterprise‑level capabilities such as dictionary binding, field encryption, data masking, dynamic DDL maintenance, sharding, multi‑datasource switching, performance logging, and data permission control, with detailed Spring Boot integration examples.
The article describes how a one‑year‑old loan/repayment service was refactored by introducing a 50‑table sharding scheme with ShardingSphere, detailing the design decisions, historical data migration, code changes, configuration files, pitfalls, and runtime synchronization strategies to achieve efficient query performance on billions of rows.
This comprehensive guide walks backend developers through practical techniques—such as proper indexing, SQL refactoring, parallel remote calls, asynchronous processing, caching layers, sharding, and monitoring—to diagnose and dramatically improve API response times and overall system throughput.
This article presents nine comprehensive performance‑optimization strategies—including load balancing, sharding, read/write separation, caching, indexing, CDN, asynchronous processing, code refinement, and algorithm improvement—aimed at boosting the efficiency and scalability of large‑scale backend systems.
This article details how Bilibili's Membership Store tackled massive traffic spikes by optimizing call chains, introducing asynchronous order processing, and implementing a sharding strategy that split databases and tables, ultimately achieving over 4000 TPS and stable performance during large‑scale promotions.
This article explains how Vitess was created to overcome MySQL leader‑follower replication limits at YouTube, detailing its sidecar VTTablet, stateless VTGate router, topology key‑value store, and scaling strategies that enable billions of users to be served reliably.
This article objectively compares NewSQL databases with middleware‑based sharding solutions, examining their architectures, distributed transaction handling, scalability, high‑availability mechanisms, storage engines, and suitability for various workloads, and provides guidance on selecting the appropriate approach based on specific system requirements.
This article shares a practical case study on how a coupon‑distribution system evolved from a single‑table design to a sharded, multi‑database architecture with query optimization, ElasticSearch integration, and read/write splitting to handle growing data volume, complex queries, and increasing request traffic.
Consistent hashing replaces simple modulo‑based distribution to efficiently locate cached data across changing numbers of servers, using a hash ring and virtual nodes to ensure balanced load, minimize data movement, and improve reliability in distributed caching, load balancing, and database sharding scenarios.
This article examines the open‑source XXL-JOB distributed task scheduler, explaining its lightweight architecture, time‑wheel based scheduling, consistent‑hash routing, sharding mechanisms, and includes annotated source code snippets to illustrate key implementation details for backend developers.
This article summarizes a live session where Shawn explains how understanding business scenarios drives effective OpenGemini database design, and provides comprehensive best‑practice guidance on library and table design, data ingestion, query optimization, and performance tuning for time‑series workloads.
This article explains database sharding concepts, reasons for using it, various sharding strategies, and demonstrates implementation with ShardingSphere, including configuration and Java code examples, while also offering additional resources for advanced architecture and interview preparation.
This article examines Pinterest's evolution through four development phases, the core technologies it adopted, and how it transitioned from complex clustering to a simpler sharding architecture to achieve scalable, high‑availability data storage.
This article outlines a comprehensive backend architecture for community platforms and billion‑level e‑commerce systems, covering microservice decomposition, DDD modeling, caching strategies, Redis clustering, message‑queue decoupling, database sharding, read‑write separation, distributed transaction approaches, multithreaded data migration, and massive counting techniques.
This article provides a comprehensive step‑by‑step guide to setting up MongoDB replica sets and sharded clusters, explaining the architecture, member roles, configuration files, initialization commands, and operational procedures for ensuring data redundancy, high availability, and horizontal scaling.
This article objectively compares NewSQL distributed databases with traditional middleware‑based sharding solutions, examining their architectures, distributed transaction support, high availability, scaling, storage engines, SQL capabilities, maturity, and suitability for various workloads, to help architects decide which approach best fits their needs.
This article objectively compares NewSQL databases with middleware‑based sharding solutions, examining their architectural differences, transaction handling, scalability, high‑availability mechanisms, storage engines, and maturity to help readers decide which approach best fits their specific workload and business requirements.
This article objectively compares NewSQL databases and middleware‑plus‑sharding architectures, examining their core principles, distributed transaction handling, high‑availability mechanisms, scaling and sharding strategies, SQL support, storage engines, and maturity to help engineers decide which solution fits their workload.
This article walks you through Redis’s architectural journey—from a simple single‑node setup, through persistence mechanisms, master‑slave replication, Sentinel‑driven automatic failover, and finally sharding with Redis Cluster—explaining each component’s purpose, trade‑offs, and how they collectively boost performance and reliability.
This comprehensive tutorial walks you through MongoDB replica set concepts, member roles, primary election, configuration commands, as well as sharding fundamentals, shard key selection, cluster deployment, balancer management, and practical code examples for building a production‑grade database cluster.
This article objectively compares middleware‑based sharding with modern NewSQL distributed databases, examining their architectural differences, performance, transaction support, scalability, high‑availability, and operational considerations, to help practitioners decide which approach best fits their workload and organizational constraints.
This article explains Redis Cluster's sharding architecture, demonstrates how to configure multiple Redis nodes on different ports, shows commands for creating and testing the cluster, and illustrates failover behavior, highlighting its scalability and high‑availability advantages over Sentinel mode for large‑scale data workloads.
The article objectively compares NewSQL distributed databases with traditional middleware‑based sharding solutions, examining their architectural differences, distributed transaction support, performance, scalability, high‑availability mechanisms, storage engines, and practical suitability for various application scenarios.
This article provides a comprehensive, top‑down and bottom‑up explanation of ElasticSearch’s core architecture, detailing nodes, shards, Lucene segments, inverted indexes, stored fields, document values, caching, query processing, routing, and scaling considerations for efficient search operations.
Facing a 50‑million‑row financial transaction table that grew 6 million rows per month, the team analyzed the problem, set clear split goals, evaluated sharding middleware, designed a custom pagination algorithm, built a hybrid data‑migration plan, and executed a three‑stage rollout to safely replace the monolithic table with multiple 10‑million‑row shards.
This article presents a detailed case study of optimizing a distributed batch processing system by replacing static shard‑key concurrency with a dynamic task‑grabbing mechanism, dramatically reducing execution time from several hours to under fifteen minutes while maintaining stable resource usage.
The article presents a comprehensive case study of Ctrip's flight order itinerary system, detailing its background challenges, design goals, modular architecture, data aggregation strategies, sharding and Redis caching optimizations, and the adoption of Java structured concurrency to achieve high performance, reliability, and scalability.
This article introduces sharding-mybatis, a lightweight, non‑intrusive sharding library that seamlessly integrates with Spring Boot and MyBatis, explains why it may be preferable to Sharding‑Sphere, lists its features, provides a GitHub link, and offers detailed usage instructions with code examples.
This article examines how database middleware tackles the demanding needs of large‑scale internet services by providing centralized connection‑pool management, transparent read‑write splitting, diverse load‑balancing algorithms, sharding support, automatic failover, security controls, comprehensive monitoring, and flexible backup‑recovery mechanisms.
The article examines Oracle’s evolution from RAC‑based sharding to the 23c Global Distributed Database, highlighting its RAFT replication, cross‑region capabilities, compliance benefits, and performance advantages that position it as a true multi‑cloud, multi‑location solution for modern OLTP workloads.
This article presents a real‑world case study of optimizing massive batch‑processing tasks in an ad‑platform by applying query‑level improvements, cursor‑based pagination, shard‑aware batch updates, JVM‑tuned garbage collection, and distributed load‑balancing, ultimately reducing CPU usage from 80% to under 2% and cutting query‑per‑minute volume from millions to a few thousand.
This article explains how to build an efficient in‑memory cache in Go by starting with a basic map, adding read‑write locks for concurrency, reducing lock contention through sharding, minimizing GC overhead with a ring buffer, and finally using the high‑performance BigCache library.
This article explains how to select appropriate sharding keys based on business requirements and data access patterns, describes vertical and horizontal partitioning methods, and presents common techniques such as equal‑value, index, and gene methods along with key generation strategies like auto‑increment, UUID, Snowflake algorithm, and open‑source middleware.
This article details how Bilibili’s membership‑purchase platform tackled massive traffic spikes by redesigning the order call chain, introducing concurrent and asynchronous processing, and applying a sharding strategy that split databases and tables, ultimately boosting latency performance and supporting over 4,000 TPS during peak sales.
This guide explains how to design a high‑concurrency, high‑throughput internet architecture that ensures database high availability with double‑master sync and virtual IPs, and how to horizontally shard and smoothly expand the cluster in seconds using configuration changes, reloads, and cleanup steps.
The article explains why traditional MySQL sharding is problematic, introduces the open‑source distributed database TiDB, details its architecture, core features, use cases, performance benchmarks against MySQL, and provides guidance on when to adopt TiDB for large‑scale workloads.
This article details the end‑to‑end design of a payment system, covering transaction flow, horizontal and vertical pre‑optimizations, task scheduling, sharding strategies, data structures, high‑availability mechanisms such as channel isolation and Hystrix, and future planning for dynamic scaling and intelligent routing.
This article explains how to assess MySQL table size through table capacity, disk usage, and instance resources, illustrates the impact of B+‑tree index depth on query speed, and presents three practical solutions—table partitioning, database sharding, and hot‑cold data archiving—to improve performance.
This article introduces the concept of database sharding, explains how sharding middleware improves scalability and performance, and reviews six popular middleware options—ShardingSphere, MyCAT, TDDL/DRDS, Atlas, Vitess, and Oceanus—helping engineers choose the right tool for billion‑row datasets.
Bilibili’s membership purchase system was re‑engineered by refactoring serial service calls into a responsibility‑chain with parallelism, moving weak dependencies to asynchronous queues, and implementing sharding across 256 tables, boosting peak throughput from ~600 QPS to over 4,000 TPS and eliminating latency incidents during massive promotional traffic spikes.
This article outlines why databases become slow, introduces a four‑layer optimization framework, and presents eight practical solutions—including data reduction, archiving, sharding, caching, and read‑replica strategies—to help engineers systematically diagnose and resolve the majority of performance bottlenecks.
This article introduces the essential concepts of sharding architecture—including data partitioning, logical and physical tables, sharding keys, routing strategies, SQL rewriting, distributed primary keys, data masking, transactions, migration, and shadow databases—providing a comprehensive guide for building high‑performance, horizontally scaled database systems.
This article outlines eight comprehensive database optimization approaches—reducing data volume, using space for performance, selecting appropriate storage systems, and related techniques such as data archiving, intermediate tables, serialization, sharding, distributed caching, read‑write splitting, and CQRS—to help backend engineers systematically address common performance bottlenecks.
This article examines the current landscape of Chinese databases, compares centralized and distributed architectures, outlines when distributed solutions are truly needed, provides performance test data, and offers practical advice on sharding, SQL design, and avoiding cross‑node bottlenecks.
To import one billion 1 KB log records stored in HDFS or S3 into MySQL as quickly as possible, the article analyzes constraints, evaluates single‑table limits, proposes sharding, batch inserts, storage‑engine choices, file‑reading strategies, task coordination, reliability mechanisms, and concurrency control.
This article examines the trade‑offs between centralized and distributed OLTP databases, presents industry usage statistics, performance benchmarks, practical questions for migration, and detailed guidance on sharding, SQL design, and operational considerations to help decide when a distributed solution is truly needed.
This article explains why database sharding is needed, the problems caused by oversized databases and tables, and how to apply vertical and horizontal splitting using MyCat, including configuration examples, splitting strategies, and testing procedures.
This tutorial walks you through adding MongoDB dependencies to a Spring Boot project, configuring the connection, defining entity and repository classes, implementing service and controller layers, and optionally setting up sharding or replica‑set architectures for scalable data management.
This article explains the principles and practical designs of high‑performance database clusters, covering read/write separation, master‑slave replication lag, distribution mechanisms, sharding (both vertical and horizontal), business‑level partitioning, and the role of NoSQL technologies such as key‑value stores, document databases, columnar databases, and full‑text search engines.
This article presents five expansion strategies—shutdown, write‑stop, log‑based, dual‑write, and smooth 2N—detailing step‑by‑step procedures for MariaDB installation, master‑master replication, dynamic data‑source configuration, and Keepalived high‑availability setup, enabling seamless horizontal scaling and minimal service disruption for large‑scale databases.
The article explains that while MySQL foreign keys ensure data consistency and integrity, they introduce performance overhead, lock contention, and scalability issues—especially in high‑concurrency and sharded environments—leading many large internet firms to forego them in favor of application‑level solutions.
This article explains how to evaluate MySQL table size, understand B+‑tree limitations, and improve query performance by using table partitioning, horizontal/vertical sharding, and hot‑cold data archiving, while also comparing the trade‑offs of each approach for practical selection.
This article analyzes the constraints of loading one billion 1 KB log records from distributed storage into MySQL, evaluates single‑table limits, proposes batch‑insert, sharding, storage‑engine, file‑reading, and distributed‑task coordination techniques to achieve high‑speed, ordered, and reliable data ingestion.
When MySQL tables grow to millions of rows, insert and query latency increase, schema changes become costly, and irrelevant data bloats storage, so this guide evaluates table size, explains why B+‑tree depth hurts performance, and presents partitioning, sharding, and hot‑cold archiving solutions with practical SQL examples.
This article explains MongoDB's three cluster architectures—deprecated Master‑Slave replication, the highly available Replica Set with automatic failover, and horizontal scaling Sharding—detailing their roles, data flow, advantages, limitations, and operational considerations for building reliable, scalable databases.
The article outlines eBay’s key scalability best practices—including functional partitioning, horizontal sharding, eliminating distributed transactions, aggressive asynchronous decoupling, intelligent caching, and pervasive virtualization—to illustrate how large‑scale web systems can achieve linear or sub‑linear growth while maintaining availability and performance.
This article explains database sharding concepts, compares vertical and horizontal partitioning, details ID‑range and modulo sharding methods, discusses the read‑amplification issue caused by non‑shard queries, and presents solutions such as auxiliary index tables, Elasticsearch integration, and TiDB as alternatives.
This article presents a hands‑on walkthrough of implementing in‑database horizontal partitioning by month for a high‑volume billing table, covering the business background, design decisions, stored‑procedure table creation, dynamic table selection in MyBatis, and query strategies such as serial‑number lookup and date‑range retrieval.
This article breaks down real‑world high‑concurrency strategies—horizontal scaling, caching, Elasticsearch, sharding, message‑queue smoothing, and cellization—explaining when each applies, their trade‑offs, and practical tips for building scalable, reliable backend services.
This article explains why relational databases hit performance bottlenecks at large scale, introduces vertical and horizontal sharding techniques, compares their pros and cons, discusses common challenges such as distributed transactions, joins, pagination and global key generation, and offers practical solutions and middleware options.
MongoDB is an open‑source, C++‑based NoSQL document database that stores data as BSON documents in collections, offering schema‑less modeling, embedded structures, high scalability through sharding, and a rich set of CRUD commands for modern application needs.
The article examines a 7‑node Elasticsearch cluster with 500 million documents, uncovering excessive heap usage, high OS memory pressure, numerous deleted documents, large translog, low query‑cache hit rate, and an over‑sharded design, then offers concrete tuning and redesign recommendations to restore performance.
