This article defines distributed databases, examines their external traits such as write‑heavy, low‑latency, massive concurrency, massive storage and high reliability, explores internal architectures like client‑side sharding, proxy middleware and unit‑based designs, compares them with Amazon Aurora, and summarizes key takeaways.
This article explains how Qunar’s ticket ancillary service upgraded from a single-database architecture to a sharded one without downtime, detailing dual-write, transaction handling, mapping-key routing, diff verification, the challenges encountered, and a component-based solution that makes future migrations reusable.
The article explains that the core of the programmer's 35‑year crisis is the rapid pace of technological change outstripping learning speed, illustrated with backend sharding, frontend toolchain shifts, and AI advances, and offers practical advice to broaden skills beyond narrow tech stacks.
This article analyzes a 7‑node Elasticsearch cluster with 500 million documents, revealing excessive shard count, high heap and OS memory usage, large translog, low query‑cache hit rate, and heavy I/O, and offers concrete recommendations on mapping, sharding, JVM tuning, and resource management to restore performance.
This article walks through Redis's evolution from a simple single‑instance cache to a highly available, high‑performance cluster, explaining persistence mechanisms (RDB, AOF, hybrid), master‑slave replication, Sentinel automatic failover, and sharding strategies with concrete examples and trade‑offs.
This article explains the AKF Cube methodology for scaling distributed systems, detailing how horizontal replication (X axis), functional decomposition (Y axis), and user‑based sharding (Z axis) can be combined to boost performance while balancing implementation costs.
This tutorial walks through the fundamentals and architecture of ShardingSphere, compares it with other sharding tools, and demonstrates how to implement database sharding in a Spring Boot project using both YML configuration and pure Java code, including detailed code snippets and deployment tips.
This article explains how to smoothly migrate a legacy single‑database system to a sharded architecture by using dual‑write, mapping‑key routing, custom transaction handling, and offline/real‑time diff checks, while detailing the underlying MyBatis integration and component‑based design.
This article chronicles JD Baitiao’s journey from early MySQL and NoSQL solutions through DBRep to the adoption of Apache ShardingSphere, highlighting the technical motivations, decoupling strategies, performance comparisons, and the broader Database Plus vision for scalable, stable financial‑grade data architectures.
The DBLE 3.23.04.0 release introduces six new or refactored features, fixes over five critical community issues and more than a hundred bugs, adjusts TCP keep‑alive parameters, adds experimental configuration support, and provides detailed LTS version updates and incompatibility notes for users.
This article explores the challenges of distributed join queries, detailing MySQL sharding‑jdbc join implementation, routing strategies, and code examples, then examines Elasticsearch‑SQL join capabilities, various join algorithms, and practical considerations for using nested types, offering insights for optimizing performance in distributed data environments.
This article describes the challenges of migrating a long‑running monolithic database to a sharded architecture, presents a two‑stage smooth migration strategy using double‑write, diff verification, and mapping‑key techniques, and details the componentized design, core implementation, and configuration of a reusable sharding framework built on MyBatis and Spring.
Facing a 50‑million‑row financial transaction table, the team implemented horizontal sharding with sharding‑jdbc, designed migration and pagination strategies, tackled multi‑datasource transaction issues, and outlined a phased rollout, providing practical insights for large‑scale MySQL table splitting and backend system stability.
This article recounts how a finance team tackled a 50‑million‑row MySQL table by analyzing system bottlenecks, defining splitting goals, selecting sharding‑jdbc, addressing multi‑datasource transactions and pagination, designing data migration strategies, and executing a staged rollout to ensure stability and performance.
This article introduces the fundamental concepts of database sharding, including data fragmentation, nodes, logical and physical tables, broadcast and single tables, sharding keys, strategies, algorithms, binding tables, SQL parsing, routing, rewriting, execution, result merging, distributed primary keys, data masking, distributed transactions, migration, and shadow databases, providing a comprehensive guide for designing and operating a sharded architecture.
This article details how a finance team tackled a 50‑million‑row MySQL transaction table by designing a horizontal sharding strategy with Sharding‑JDBC, addressing performance bottlenecks, multi‑source transaction handling, pagination across shards, and a phased data migration and rollout plan.
The article details how a finance team tackled a 50‑million‑row MySQL transaction table by analyzing the pre‑split problems, defining split goals, choosing sharding‑jdbc, addressing multi‑datasource transaction and pagination issues, designing a phased migration and rollout plan, and summarizing lessons learned.
This article compares common database sharding strategies—range sharding, mapping tables, ID modulo, hash‑based sharding, and consistent hashing—explaining their implementation complexity, data distribution characteristics, scalability concerns, and practical trade‑offs with code examples.
This article explains common business scenarios such as large‑table splitting, cross‑database migration, and data synchronization, then compares loss‑bearing and loss‑less migration strategies, and provides detailed step‑by‑step implementation guidance for smooth dual‑write migration and real‑world tooling.
This comprehensive guide explains MongoDB cluster architecture, component roles, common use cases, monitoring commands, essential maintenance operations, data migration steps, troubleshooting of typical production issues, and practical optimization recommendations for high‑performance deployments.
This technical guide explains how to scale a sharded database horizontally by introducing five expansion schemes—including shutdown, write‑stop, log‑based, dual‑write, and smooth 2N approaches—covers MariaDB installation, master‑master replication setup, dynamic data‑source configuration with ShardingJDBC, and detailed Keepalived high‑availability configuration for seamless service continuity.
This article explains how to assess MySQL table size, identify performance issues caused by massive data volumes, and presents three practical solutions—table partitioning, database sharding, and hot‑cold data archiving—along with detailed SQL examples and considerations for implementation.
This article explains the concepts of database sharding, compares vertical and horizontal partitioning, discusses ID‑range and modulo sharding methods, analyzes the read‑amplification problem caused by non‑sharding keys, and presents practical solutions such as auxiliary index tables, Elasticsearch integration, and TiDB adoption.
This article explains why large MySQL tables cause slow queries, analyzes table size, disk usage, and instance capacity, and presents three practical solutions—table partitioning, database sharding (horizontal/vertical), and hot‑cold data archiving—helping developers choose the appropriate strategy for their workloads.
This article explains the concepts, strategies, and practical scenarios for MySQL database partitioning, covering both table splitting (vertical and horizontal) and database splitting (vertical and horizontal), with examples, advantages, and when to apply each technique to handle large-scale data workloads.
This article explains how to assess MySQL table size, why massive tables degrade performance due to B+‑tree depth, and presents three practical solutions—table partitioning, database sharding, and hot‑cold data archiving—along with guidance on selecting the appropriate approach.
This article explains how Apache Doris implements data sharding, partitioning (both list and range, static and dynamic), batch partition creation, and the Auto Bucket feature, providing detailed syntax, algorithms, and performance benefits for large‑scale distributed databases.
This comprehensive guide explores essential backend techniques—including lock‑free programming, zero‑copy I/O, efficient serialization, pooling, concurrency, async processing, caching strategies, sharding, storage optimizations, and queue mechanisms—to build high‑performance, scalable services while highlighting practical code examples and real‑world trade‑offs.
This guide walks developers through transforming a single‑node Redis cache into a stable, high‑performance cluster by explaining data persistence options (RDB, AOF, hybrid), master‑slave replication, Sentinel‑driven automatic failover, and both client‑side and proxy‑based sharding for horizontal scalability.
The article explains the limitations of standalone MySQL for large‑scale workloads, explores sharding, read/write separation, and middleware approaches, and argues that modern distributed databases and cloud‑native solutions are the sustainable path forward for data‑intensive applications.
This comprehensive guide explains what Elasticsearch does, its underlying Lucene technology, core concepts such as clusters, shards, replicas, mapping, indexing and storage mechanisms, and provides practical performance‑tuning advice for building and operating a robust distributed search engine.
The article explains how single‑instance MySQL struggles with huge tables, resource‑starved databases, and high read traffic, and argues that sharding middleware is being replaced by modern distributed databases such as TiDB, OceanBase, Aurora, and PolarDB as the sustainable scaling solution.
When order_id is not the sharding key, querying a sharded order table can require scanning every shard, but three techniques—data redundancy, an index table, and embedding shard information into the order_id (gene method)—enable fast, single‑shard lookups.
The article explains why auto‑increment primary keys are unsuitable for sharded databases, recommends globally unique ordered identifiers, and details index and global‑table strategies—including redundant tables, index tables, and embedding shard information in keys—to achieve efficient single‑shard queries.
This article provides a comprehensive tutorial on configuring MySQL master‑slave replication with Docker, explains the roles of binlog and redo log, describes two‑phase commit, and demonstrates how to use Sharding‑JDBC (ShardingSphere) for read‑write splitting and horizontal sharding in a Spring Boot application, complete with configuration files and code examples.
This article presents a practical guide to implementing monthly sharding for high‑volume billing data using MySQL stored procedures, dynamic table creation, and SpringBoot‑MyBatis integration, covering schema design, code examples, and query strategies for inserting, retrieving, and managing partitioned tables.
This article explains why and when to split databases and tables, how to choose sharding keys, various sharding strategies such as range, hash and consistent hash, handling cross‑node joins, pagination, distributed IDs, middleware choices, and step‑by‑step zero‑downtime migration techniques.
This article explains Elasticsearch’s core concepts—including inverted indexes, shard architecture, node roles, and the detailed write‑read‑search workflow—so readers can grasp how the system achieves near‑real‑time search and reliable data storage.
The article explains how traditional single‑node MySQL faces scalability challenges such as massive tables, resource exhaustion, and read‑heavy workloads, and argues that adopting sharding, middleware, or modern distributed databases like TiDB, OceanBase, Aurora, and PolarDB is the inevitable path forward for robust data management.
This article explains the end‑to‑end design of a pickup code system—including simple single‑table implementation, sharding‑aware multi‑database strategies, code generation algorithms, concurrency handling with distributed locks, and practical solutions to common pitfalls—targeted at SaaS e‑commerce back‑ends.
This article explains Elasticsearch's inverted index structure, sharding and replica concepts, cluster node roles, and the detailed write, read, search, delete, and update processes, illustrating how data moves from memory to disk and becomes searchable in near real‑time.
This article explains the core concepts of Elasticsearch, including how its inverted index works, the structure of term dictionaries and posting lists, shard and replica configuration, cluster node roles, the detailed write, refresh, flush, and merge processes, as well as how search queries are executed across distributed shards.
This article explains the core concepts of Elasticsearch, including the structure and purpose of inverted indexes, how shards and replicas are organized in a cluster, and the detailed workflows for writing, reading, searching, and deleting documents within a distributed environment.
This article explains the fundamentals of structured and unstructured data, introduces Lucene’s inverted index, and details how Elasticsearch builds on Lucene to provide distributed, near‑real‑time search with concepts such as clusters, shards, replicas, routing, and performance optimizations.
This article explains how Redis achieves high availability through master‑slave replication, automatic failover with Sentinel, and true sharding with Redis Cluster, while also comparing popular sharding solutions such as client‑side hashing, Twemproxy, and Codis.
When order_id is not the sharding key, querying a sharded order table can require scanning every shard, but three design patterns—data redundancy, an index table, and gene‑based sharding—enable fast single‑shard lookups while balancing storage and complexity.
This article presents a step‑by‑step practice of migrating massive billing data to 32 sharded databases with Sharding‑Proxy, covering background, objectives, four migration solutions, detailed proxy installation and configuration, debugging, migration workflow, data validation, common issues and their resolutions.
This article explains MySQL's theoretical row limits, why performance degrades long before reaching them, and how horizontal and vertical database sharding can distribute massive datasets across multiple instances to improve load handling and enable seamless scaling.
This guide explains how to scale a sharded database horizontally by adding new nodes, handling data migration with stop‑service, stop‑write, log‑based, dual‑write, and smooth 2N strategies, and implementing high‑availability using MariaDB double‑master replication and Keepalived, complete with configuration examples and code snippets.
This article compares the architectures of OceanBase and TiDB, explaining how their sharding and compute‑storage separation designs affect high availability, latency, and workload suitability, and offers guidance on selecting the right distributed database for complex enterprise applications.
This article explains why high‑concurrency big‑data systems need database sharding, describes table‑level partitioning methods (modulus and range), shows how to implement them with MySQL and Java code, compares vertical and horizontal database splitting, and evaluates middleware solutions such as Sharding‑JDBC and Mycat.
This article explains how to handle billions of order records by classifying data into hot and cold tiers, using MySQL sharding with database‑and‑table partitioning, storing cold data in Elasticsearch and Hive, and outlines combined routing formulas for scalable backend architecture.
This article outlines eBay's key scalability best practices—including functional decomposition, horizontal sharding, avoiding distributed transactions, asynchronous decoupling, virtualization, and intelligent caching—to demonstrate how large‑scale web systems can achieve linear resource growth and high availability.
This article presents the design, implementation challenges, and optimization strategies of an intelligent auto‑placement platform, covering background analysis, distributed task scheduling with XXL‑Job, sharding with ShardingSphere, caching via Caffeine, message‑queue integration using Kafka, and the resulting performance gains.
The article explains how to select and implement routing keys for database sharding in an order platform, covering time‑based and business‑related keys, user and merchant scenarios, hash‑modulo strategies for single and multi‑database environments, and the overall data flow.
This article explains the evolution from single‑node MySQL databases to master‑slave replication, various replication modes, high‑availability strategies, and both vertical and horizontal sharding techniques, while discussing the associated challenges such as distributed transactions, routing, and operational complexity.
The article explains how to choose and implement routing keys for database sharding in an order platform, covering time‑based and business‑related keys, user and merchant scenarios, hash‑mod routing strategies, and the complete data flow with diagrams.
This article examines the challenges of sorting queries in distributed databases, outlines the limitations of in‑memory proxy sorting, and presents a step‑by‑step optimized approach that leverages per‑shard sorting, disk‑based buffering, and priority‑queue merging to reduce memory pressure and I/O overhead.
The article explains how to choose and implement routing keys for database sharding in an online food ordering system, covering supported scenarios, user‑side and merchant‑side keys, single‑ and multi‑database strategies, and the impact on data distribution and query efficiency.
This article explains why sharding tables improves read/write speed, then compares three pagination strategies—global query, no‑skip pagination, and two‑phase query—detailing their SQL rewrites, execution steps, performance trade‑offs, and practical examples.
This article explains the background, challenges, and design principles of horizontal scaling for TDSQL, detailing its architecture, scaling process, shard‑key selection, high availability, distributed transactions, performance optimization, practical case studies, and a Q&A section.
This article explains the background, simple single‑table and complex sharding designs, global versus store‑unique strategies, and practical solutions such as distributed locking and retry mechanisms for generating and managing pickup codes in a SaaS e‑commerce system.
Effective index and primary key design in distributed database architectures requires using globally unique, ordered identifiers instead of auto‑increment keys, embedding shard information in keys, employing auxiliary index tables, and handling global tables to ensure queries target a single shard, thereby preserving scalability and performance.
This article walks through a real‑world large‑scale MySQL sharding project, covering business refactoring, storage architecture design, data migration, incremental upgrades, best‑practice tips, and stability safeguards, while sharing concrete steps, pitfalls, and lessons learned from start to production rollout.
This article examines why single-node databases no longer meet modern internet workloads, explores MySQL replication models (master‑slave, asynchronous, semi‑synchronous, group replication), discusses high‑availability solutions such as MHA, MGR and Orchestrator, and outlines vertical and horizontal sharding techniques along with their trade‑offs.
This article explains eight core software architecture patterns—from single‑database single‑application to microservices, multi‑level caching, and multi‑datacenter deployments—detailing their designs, typical use cases, advantages, and drawbacks to help engineers choose the right approach for scalable, high‑performance systems.
This guide explains how to install mlaunch and use it to rapidly set up MongoDB single nodes, replica sets, and sharded clusters, providing step‑by‑step command examples, operational commands, and reference links for efficient testing environments.
This article analyzes the performance and storage problems of the Windows version of WeChat, explains why message growth causes slowness, large file size, and corruption risk, and presents a multi‑pronged solution involving sharding, indexing, payload compression, and robustness improvements that halve database size and boost I/O performance by about ten percent.
This article explores the fundamental concepts and common pitfalls of distributed systems—including CAP and BASE theories, message‑queue reliability issues, Redis sentinel challenges, sharding strategies, and distributed transaction models—offering practical guidance to avoid costly failures in real‑world deployments.
This article explains ClickHouse’s MPP columnar design, complete DBMS capabilities, columnar storage, vectorized execution, multi‑master architecture, real‑time queries, sharding, and performance‑focused hardware and algorithm choices that together deliver its superior speed.
This article provides a comprehensive overview of distributed system fundamentals—including CAP and BASE theories—while detailing typical pitfalls and mitigation strategies for message queues, distributed caches, database sharding, and distributed transactions, helping engineers decide when and how to adopt distributed architectures.
This article details the architectural evolution of a price‑increase coupon system used in a C2B digital‑product recycling business, covering its initial experimental phase, platformization, sharding, intelligent recommendation, Elasticsearch integration, and stability enhancements through NoSQL and read‑replica strategies.
This article explains the fundamentals of distributed transactions, including ACID properties, consistency models, sharding strategies, and the two‑phase, three‑phase, and TCC protocols, while discussing CAP and BASE theories and the challenges of implementing reliable distributed databases.
This article explains why single‑node Redis suffers from single‑point failures, describes the master‑slave replication model, details the Sentinel automatic failover mechanism, compares various sharding solutions like client‑side sharding, Twemproxy, Codis, and outlines the features and deployment considerations of Redis Cluster.
The article explains why simply adding hardware cannot keep up with growing database workloads and presents a three‑step approach—business isolation, horizontal sharding, and advanced reporting using PolarDB‑X—to achieve high concurrency, elastic capacity, and efficient caching for modern high‑traffic applications.
This article explains the three aspects of database sharding—database‑only, table‑only, and both—detailing when to apply each, how to choose sharding keys, common algorithms, ID generation strategies, available open‑source tools, and the new challenges introduced by sharding.
This article explains why and how to apply database sharding, introduces key terminology, compares global ID generation methods such as auto‑increment, UUID, COMB and Snowflake, outlines sharding algorithms, discusses challenges like distributed transactions, and presents practical expansion and implementation solutions.
This article details the design, evolution, and scaling strategies of a price‑increase coupon system for a C2B digital product recycling platform, covering its initial experimental phase, platformization, sharding‑JDBC implementation, intelligent coupon recommendation, Elasticsearch integration, and operational optimizations for high‑throughput stability.
This article explains how Redis tackles single‑point failures with master‑slave replication, introduces Sentinel for automatic failover, compares client‑side and proxy sharding solutions like Twemproxy and Codis, and details the native Redis Cluster architecture for true distributed storage and high availability.
This article explains the DBLE split feature that partitions large mysqldump files according to sharding configuration, provides command syntax and usage examples, and presents a performance test showing that split‑based import can be up to 18 times faster than direct DBLE import while preserving data correctness.
This article explains why large‑scale systems need data partitioning beyond replication, defines partitions and sharding, compares key‑range and hash‑based partitioning methods, discusses load‑balancing and hotspot avoidance, and shows how replication works together with partitioning to improve scalability.
This article examines the most common Chinese distributed database architectures, comparing integrated SHARDING designs like OceanBase with separated compute‑storage models such as TiDB and SQL‑proxy based systems, highlighting their inherent trade‑offs, performance challenges, and practical considerations for choosing the right solution.
This article explains the fundamentals of structured and unstructured data, introduces Lucene's inverted index, describes Elasticsearch's distributed cluster architecture, node roles, sharding and replication mechanisms, indexing workflow with refresh and translog, storage segment model, and provides practical performance‑tuning recommendations.
This article introduces MyBatis‑Mate, an official MyBatis‑Plus extension that provides enterprise‑grade capabilities such as dictionary binding, field encryption, data masking, dynamic table maintenance, audit logging, sharding, multi‑datasource switching, and customizable data‑scope handling, all illustrated with code examples.
This article explains Redis high‑availability strategies, covering basic hash sharding, the advantages of consistent hashing, client‑side versus proxy‑based partitioning, master‑slave replication, and the Sentinel failover mechanism, with diagrams illustrating node addition, removal, and failover decision processes.
This article provides a comprehensive overview of ElasticJob‑Lite, a distributed scheduling solution for high‑concurrency environments, covering its architecture, ZooKeeper‑based registration and sharding mechanisms, core listener components, key code examples, and the UI console deployment.
This article provides a comprehensive guide to MySQL index optimization, covering selectivity calculation, prefix length, composite index rules, EXPLAIN analysis, common slow‑query patterns, and practical sharding and partitioning techniques to improve query performance.
This article explains core cluster concepts, the benefits of building database clusters, classifies common cluster types, and compares scalable architectures such as Oracle RAC, MySQL Cluster, sharding, CAP/BASE theory, and cross‑database transaction strategies for high availability and performance.
This article explores eight common backend architecture patterns—from simple single‑database setups to microservices, caching, sharding, elastic scaling, and multi‑region deployments—detailing their design principles, typical use cases, advantages, drawbacks, and practical implementation steps.
This comprehensive guide explores MongoDB’s core features—including schema flexibility, high availability, sharding, storage engine internals, indexing, and performance tuning—while providing practical examples, configuration tips, and best‑practice recommendations for developers and architects seeking to efficiently deploy and operate MongoDB in production environments.
This guide walks through installing a Redis cluster with three masters and three slaves using local ports, explains slot allocation, key hashing, gossip communication, failover, node addition, resharding, and best practices for high availability, while providing practical commands and configuration examples.
This article explains essential techniques for building high‑concurrency systems—including load balancing algorithms, microservice decomposition, caching strategies, database sharding, message‑queue integration, and CDN usage—to help engineers design scalable architectures and advance their careers.
The article introduces eight widely used software architecture design patterns—including single‑database, content distribution, query separation, microservices, multi‑level caching, sharding, elastic scaling, and multi‑datacenter deployment—explaining their typical structures, suitable scenarios, and the key pros and cons of each.
In just 13 days Bilibili transformed a simple MySQL‑based S3 prototype into the BOSS distributed object storage system by separating metadata and data, adding an RPC abstraction layer, implementing two‑level sharding, switching to RocksDB, and deploying a three‑replica, multi‑zone high‑availability architecture.
This comprehensive guide explains what Elasticsearch does, its underlying Lucene engine, core concepts like clusters, shards, replicas, mappings, and provides practical steps for installation, configuration, indexing, storage mechanics, and performance optimization.
This article chronicles the step‑by‑step evolution of a ride‑hailing order service, detailing how a single SQLServer database was optimized, split by business domain, migrated to MySQL, and finally sharded with a custom client‑side component, while leveraging read/write separation, caching, MQ, and data‑exchange platforms to sustain massive traffic.
This article explains why databases become slow, introduces a four‑layer thinking model, and presents eight practical solutions—including reducing data volume, leveraging space for performance, and selecting the right storage system—to help backend engineers resolve 80‑90% of common performance problems.
This article explains MySQL's lack of a hard row limit, the industry rule of thumb around 5 million rows, how InnoDB buffer size and hardware affect performance, and demonstrates a simple insertion test that reveals slowdown when a table grows too large.
This article outlines a systematic approach for backend engineers to identify root causes of database slowness and presents eight concrete optimization techniques—ranging from data reduction and space‑for‑performance trade‑offs to appropriate storage system selection—while explaining when each method is most effective and what trade‑offs to consider.
This article explains the fundamentals of distributed transactions, the ACID properties, various consistency models, database sharding strategies, and the two‑phase, three‑phase, and TCC commit protocols, while also discussing CAP, BASE, and practical challenges in large‑scale systems.
