This article explores common e‑commerce backend challenges such as duplicate orders, order snapshots, cart storage, inventory oversell, logistics updates, account balance consistency, read‑write splitting, hot‑cold data separation, and sharding, and provides concrete design patterns and SQL examples to solve them.
This article explains the fundamentals of distributed transactions, the ACID properties, various consistency models (strong, weak, eventual), sharding strategies (vertical and horizontal), the CAP and BASE theories, and the practical implementations of two‑phase, three‑phase, and TCC commit protocols, highlighting their advantages and drawbacks.
This article recounts the end‑to‑end experience of planning, executing, and reviewing a database sharding project for a high‑traffic order platform, detailing pre‑implementation risk assessment, the detailed testing lifecycle, gray‑release strategies, performance validation, and post‑mortem lessons learned.
This article provides a comprehensive overview of Elasticsearch, explaining its underlying Lucene technology, data modeling, cluster architecture, shard and replica mechanisms, indexing workflow, storage strategies, refresh and translog processes, as well as practical performance and JVM tuning tips for building scalable, near‑real‑time search solutions.
This article explains why databases become slow, introduces a four‑layer thinking model, and presents eight practical optimization techniques—including data reduction, caching, sharding, master‑slave replication, and CQRS—along with their benefits, drawbacks, and suitable scenarios.
This article explains the fundamentals of distributed transactions, covering ACID properties, consistency models, sharding strategies, CAP and BASE theories, and compares two‑phase, three‑phase, and TCC commit protocols to illustrate how modern systems maintain data integrity across multiple nodes.
The article explains how I/O and CPU bottlenecks increase active database connections, then details various sharding techniques—including horizontal and vertical database/table partitioning, their concepts, results, scenarios, and analysis—followed by tools, implementation steps, common issues, and best‑practice recommendations.
This article outlines essential architectural practices for high‑concurrency systems, covering load‑balancing strategies, distributed microservice design, caching mechanisms, database sharding, asynchronous message queues, and CDN usage to achieve scalable and resilient backend services.
This article explains microservice fundamentals, their advantages, and how to design data architectures that support independent services, covering database-per-service versus shared databases, polyglot persistence, scaling strategies, and a detailed MongoDB case study with practical guidelines.
This article explains the concepts, methods, advantages, and drawbacks of database sharding, covering vertical and horizontal partitioning, sharding rules, global primary key generation, distributed transaction issues, cross‑node queries, pagination, data migration, and practical case studies, with code examples and diagrams.
This article provides a comprehensive overview of Elasticsearch, covering its data types, Lucene-based inverted index, cluster architecture, sharding and replication mechanisms, mapping definitions, basic usage, health monitoring, storage internals, and practical performance tuning tips for large‑scale search deployments.
This article explains the concepts of ClickHouse local and distributed tables, why writing directly to distributed tables can be problematic, and how replication, sharding, and the ReplicatedMergeTree engine work together with ZooKeeper to provide high‑availability and scalable query processing.
This article explains how to design a flexible, extensible database schema and login flow for a system with a billion users, covering multi‑credential handling, sharding strategies, hash‑based routing, and practical considerations such as password updates and caching.
This article explains database sharding fundamentals, including terminology, when to adopt sharding, various global ID generation methods such as auto‑increment, UUID, COMB, and Snowflake, different partitioning strategies, challenges like distributed transactions, and practical expansion and migration solutions.
This article introduces mybatis-mate, an official MyBatis‑Plus extension that provides enterprise‑grade features such as sharding, multi‑datasource support, data auditing, field encryption, sensitive‑data masking, automatic DDL maintenance, dynamic datasource switching, distributed transaction logging, and fine‑grained data permission control, with detailed usage examples and configuration snippets for Spring Boot projects.
This article explains Elasticsearch index concepts, index types, and provides step‑by‑step REST commands for creating, listing, deleting indices, indexing documents, querying, managing shards and replicas, and checking cluster health.
The article walks through the challenges of growing data volumes, compares master‑slave, vertical, horizontal, and sharding‑plus‑partitioning architectures, explains their pros and cons, and provides concrete guidance on selecting and implementing a suitable database solution, including middleware options and real‑world case studies.
A comprehensive collection of MySQL interview questions and answers covering storage engines, lock types, gap locks, deadlock avoidance, isolation levels, index types, covering indexes, the left‑most prefix rule, SQL tuning, master‑slave replication, latency handling, sharding challenges, and global ID generation.
This article explains Elasticsearch’s distributed cluster design, covering nodes, indices, shards, replicas, deployment models, data storage options, and the trade‑offs of different distributed system architectures for search and analytics workloads.
This article explains the fundamentals of RabbitMQ, compares normal and mirrored cluster modes, details Google’s large‑scale test setup, and walks through advanced plugins such as sharding, consistent‑hash exchange, federation, and high‑availability strategies for achieving million‑level message throughput.
This article explains how to horizontally scale RabbitMQ clusters, use sharding and federation plugins, configure mirror queues and other high‑availability features, and apply practical patterns such as confirms, retries, and delayed delivery to achieve million‑level message throughput in production environments.
This article explains how to horizontally scale RabbitMQ clusters to handle millions of messages per second by leveraging cluster modes, mirror queues, sharding plugins, consistent‑hash exchanges, federation, and high‑availability configurations, while also covering practical scenarios such as retries, delayed tasks, and Spring AMQP integration.
This article explains what database sharding (horizontal and vertical partitioning) is, why it is needed, the implementation strategies, and compares two popular solutions—Mycat middleware and Sharding-JDBC—highlighting their advantages, drawbacks, and typical use cases.
This comprehensive guide walks through MySQL performance fundamentals, real‑world testing results, slow‑query detection, index impact, pagination pitfalls, count optimizations, sharding techniques, and step‑by‑step migration plans for large tables, equipping engineers with actionable tuning insights.
This comprehensive guide covers MongoDB's core concepts—including schema‑free design, high availability, sharding, storage engine internals, indexing, and performance tuning—while providing practical code examples, configuration steps, and best‑practice tips for developers and architects looking to master the database in production environments.
This article explains Elasticsearch’s cluster and data layer architecture, covering nodes, indices, shards, replicas, mixed and tiered deployment models, storage options, and the trade‑offs of different distributed system designs for scalable systems.
The article details how Tencent Cloud’s K‑Song feed service, serving over 150 million daily users, was engineered with a read‑expansion model, cached results, auxiliary index tables, hashed sharding, write‑concern tuning, disabled chain replication, and WiredTiger and backup optimizations, achieving sub‑10 ms write latency and significantly lower CPU and slow‑query rates.
This article presents a comprehensive, practical design for an ultra‑large‑scale user center, covering microservice architecture, API separation, token generation with graceful degradation, data‑sharding strategies, password encryption, asynchronous processing, and detailed monitoring to ensure high availability, performance, and security.
When a table grows beyond ten million rows, sharding it across multiple databases improves performance, but the usual LIMIT offset, pagesize pagination breaks, leading to missing or incorrect records; this article examines why simple merge approaches fail and evaluates global, secondary query, and no‑skip paging strategies, highlighting their trade‑offs.
This comprehensive guide explains Elasticsearch index design, covering shard and replica planning, settings optimization, dynamic and static mappings, template management, lifecycle automation, and practical best‑practice recommendations for high‑performance, stable search clusters.
This article introduces Mybatis-Mate, an enterprise‑level MyBatis extension that provides sharding, data audit, sensitive‑word filtering, field encryption, dictionary binding, data permission, dynamic DDL generation and other features, and shows how to integrate and use them with practical code examples.
The article provides a detailed overview of DBLE version 3.21.10.0, covering project background, three major new features—including single‑node detach/reattach, enhanced traffic control, and default sharding mode—along with a full list of release notes, bug fixes, incompatibilities, and subsequent LTS patch updates.
This article examines common system architecture patterns—microservices, multi‑level caching, and database sharding—explaining the problems they address, how they work, their advantages and drawbacks, and practical guidance for applying them without over‑engineering.
This article introduces Elasticsearch as a distributed, high‑performance search engine, explains its cluster architecture, node roles, shard and replica mechanisms, write and refresh processes, and outlines search handling techniques such as pagination, scroll and search‑after with practical curl examples.
This article introduces MyBatis‑Mate, an official MyBatis‑Plus extension that provides enterprise‑grade capabilities such as sharding, data auditing, field encryption, dictionary binding, sensitive data masking, automatic DDL maintenance, dynamic data‑source switching, performance logging, and fine‑grained data‑scope control, all illustrated with practical code examples.
This article explains how to horizontally scale RabbitMQ clusters to achieve million‑message per second throughput, covering Google’s large‑scale test setup, sharding and consistent‑hash plugins, federation, high‑availability mirroring, reliability mechanisms, and practical deployment tips for production environments.
This article traces the evolution of database systems—from the first network and hierarchical models through relational databases, NoSQL sharding, and finally modern distributed SQL—explaining why distributed databases emerged, how they handle data distribution, consistency, SQL-to‑KV mapping, and transaction challenges.
This article explains the concepts, characteristics, and practical scenarios of database sharding—including vertical and horizontal table partitioning as well as vertical and horizontal database partitioning—helping developers decide when and how to apply each strategy to improve scalability and performance.
The article presents mybatis-mate, an enterprise‑grade MyBatis module that provides sharding, data auditing, sensitive‑word filtering, field encryption, dictionary write‑back, data permission, automatic DDL maintenance, dynamic multi‑datasource switching, and distributed transaction logging, offering developers a more agile and elegant way to handle data.
The article explains how distributed database systems appear as a single logical database but operate over a network of nodes, detailing failures, stale reads, consistency models, architecture anomalies, sharding, replication, and various read/write scenarios that illustrate the complexity of achieving strong consistency.
This article provides a step‑by‑step tutorial on configuring MySQL master‑slave replication with Docker, explains redo log and binlog mechanisms, demonstrates two‑phase commit, and shows how to use ShardingSphere‑Jdbc for read/write splitting and sharding with detailed Spring Boot examples and code snippets.
This article explains the background, design criteria, and practical implementations of horizontal database sharding—including range, hash, gene, routing‑table, and consistent‑hash methods—while discussing common pitfalls, data‑skew mitigation, and both doubling and consistent‑hash expansion techniques for MySQL clusters.
This article details how a rapidly growing e‑commerce platform migrated from a single MySQL instance to a 16‑shard architecture using Sharding‑Jdbc, introduced consistent‑hashing to mitigate data skew, leveraged ES+HBase for multi‑dimensional queries, and implemented zero‑downtime migration strategies such as dual‑write and Canal replication.
This article details the evolution of Vivo's e‑commerce coupon platform, covering its business drivers, architectural redesign, sharding strategy, multi‑mode issuance, high‑performance safeguards, and a step‑by‑step zero‑downtime migration to a standalone service.
When a MySQL table is sharded into multiple databases or tables, the usual LIMIT offset, pagesize pagination breaks, and this article examines why simple merge fails, then presents three practical approaches—global limit, secondary queries, and jump‑page restriction—highlighting their trade‑offs and limitations.
This article explains Cobar’s sharding middleware and presents an optimization technique for handling ORDER BY and LIMIT queries across multiple databases, detailing the original approach, its inefficiencies with deep pagination, the step‑by‑step improved algorithm, performance analysis, limitations, and practical applicability.
This article explains the new DBLE v3.20.10.0 feature that enables read/write splitting independently of sharding, detailing environment setup, configuration files, SQL testing, and log verification for both sharded and pure read/write split scenarios.
This article explains why MySQL queries become slow, how proper indexing, MDL locks, sharding, read‑write separation, and large‑table strategies can improve performance, and also introduces Elasticsearch and HBase as complementary technologies for search and high‑write workloads.
The article explains how Ant Group’s Logic Data Center (LDC) architecture evolved from monolithic to unit‑based distributed design, using zones, sharding, and read‑write separation to sustain a record 583,000 transactions per second during the Double‑11 shopping festival.
This guide introduces MyBatis‑Mate’s core capabilities—including dictionary binding, field encryption, data masking, automatic DDL maintenance, dynamic sharding, and data‑scope permissions—while providing Maven dependencies and complete Java code examples for integrating these features into Spring Boot applications.
The article explains how Facebook’s Shard Manager tackles large‑scale system sharding by combining stateful and stateless service deployment, consistent hashing versus sharding, fault‑as‑normal principles, replication, automated failover, load‑balancing, and elastic scaling to achieve high availability and efficient resource use.
This article describes how Ctrip's senior backend engineer designed and implemented horizontal database sharding, a service‑level proxy, dual‑read/write mechanisms, and a staged migration process to overcome order‑database bottlenecks, improve scalability, and ensure high availability for the rapidly growing international train‑ticket business.
This article outlines 58.com’s database architecture strategies—including high availability via replication and dual‑master setups, read‑performance enhancements through indexing, read replicas, and caching, consistency solutions, scalability techniques, horizontal sharding patterns, and a review of Codd’s twelve relational rules as design guidelines.
This guide explains how to build a highly available, scalable architecture for supporting hundreds of millions of users, covering decoupling, redundancy, vertical and horizontal scaling, load balancing, database replication, sharding, caching, CDN, GeoDNS, and best practices for progressive system expansion.
This article provides a comprehensive introduction to Elasticsearch, covering its underlying Lucene-based inverted index, data types, shard routing, cluster roles, discovery mechanisms, refresh and translog handling, segment merging, and practical performance and JVM tuning tips for building scalable, near‑real‑time search systems.
Version 2.3.0 of Loki introduces enhanced features such as per‑tenant custom retention policies, time‑range log deletion via the Compactor API, Prometheus‑style recording rules, a new pattern parser for LogQL, ingestion sharding for faster queries, and advanced IP‑matching syntax, all aimed at improving storage efficiency, compliance, and observability.
This comprehensive guide covers everything from MongoDB's core concepts, data model, and architecture to practical commands for creating databases, collections, indexes, sharding, replication, aggregation, and troubleshooting, providing clear answers to over fifty frequently asked questions.
This article provides a comprehensive overview of MongoDB, covering its definition, document‑oriented storage, key features, differences from relational databases, sharding and replication mechanisms, common commands, data types, indexing, aggregation, and practical usage scenarios for modern applications.
This article explains why MySQL chooses B+ trees for its indexes, covers the left‑most prefix rule for composite indexes, offers practical index‑design tips, discusses MyBatis caching, master‑slave replication, and sharding strategies for large‑scale databases.
The release of MyBatis‑Mate 1.0.1 introduces enterprise‑grade features such as dictionary binding, field encryption, data desensitization, automatic DDL maintenance, and dynamic multi‑datasource sharding, accompanied by code examples, future roadmap, and Maven dependencies for seamless integration into Spring Boot projects.
The article explains how IO and CPU bottlenecks increase active database connections, then details horizontal and vertical sharding techniques, practical tools, step‑by‑step implementation, and common pitfalls to help engineers relieve pressure on database resources.
This tutorial explains how to use Elastic Job's sharding configuration in Spring Boot to accelerate scheduled task execution, improve resource utilization, and achieve high‑availability by distributing work across multiple job instances.
This article presents a comprehensive overview of 58.com’s database architecture, covering availability through replication and dual‑master setups, read‑performance enhancements via indexing, replicas and caching, consistency strategies, rapid horizontal scaling, sharding patterns, and practical SQL considerations alongside Codd’s 12 relational rules.
This article outlines a comprehensive, step‑by‑step approach for migrating massive tables—up to billions of rows—through sharding, dual‑write, data synchronization, validation, testing, and controlled switch‑over to ensure minimal impact on production services.
The article outlines a range of techniques—including caching, read‑write splitting, vertical partitioning, sharding, cold‑hot data separation, Elasticsearch, NoSQL and NewSQL—to store and retrieve billions of user‑generated records efficiently while maintaining high throughput and low latency.
This article explains why MySQL queries become slow, how index design and common pitfalls affect performance, introduces MDL locks and large‑table strategies, then compares Elasticsearch and HBase as complementary storage and search solutions, providing practical code examples and best‑practice recommendations.
The article explains how MySQL uses different memory areas such as Thread Memory, Sharing, and InnoDB Buffer Pool, describes the impact of large user tables on query performance, and shows that when a table size exceeds the InnoDB buffer pool capacity, sharding becomes necessary.
This article explains why MySQL queries become slow, explores index-related pitfalls and optimization techniques, and then compares ElasticSearch and HBase as complementary solutions for large‑scale data and search scenarios, offering practical tips and code examples.
This article analyzes the typical evolution from monolithic to RPC-based services, explains why sharding and database partitioning cannot alone achieve unlimited scaling due to connection limits, and introduces a unitization approach that assigns each application to a specific database shard to overcome those constraints.
This article explains why MySQL queries become slow, how indexes work and fail, the impact of MDL locks, large‑table challenges, sharding and read‑write splitting strategies, then introduces Elasticsearch’s search capabilities and HBase’s column‑family storage, offering practical guidance for each technology.
This article traces the evolution of backend services from monolithic applications through RPC and sharding, explains why sharding and database partitioning cannot alone achieve unlimited scaling due to connection limits, and proposes a unitization approach that isolates database connections per service to enable true horizontal expansion.
This article analyzes common causes of slow MySQL queries such as index misuse, MDL locks, and large‑table bottlenecks, then presents practical solutions like proper indexing, sharding, read/write splitting, and evaluates when to complement MySQL with ElasticSearch or HBase for better performance.
Over four years at Ele.me, the author chronicles the transformation of the order platform from a single‑machine Python monolith to a distributed, sharded, message‑driven architecture, detailing challenges in scaling, performance testing, fault injection, Redis usage, and service separation that shaped today’s robust backend.
This article explains how to optimize Java scheduled tasks for large‑scale data synchronization by introducing phased improvements such as blacklist mechanisms, multithreaded execution with CountDownLatch, and server‑level sharding, providing practical guidelines for performance and reliability.
This article walks through Redis's architectural journey, covering single‑node basics, data persistence strategies (RDB and AOF), master‑slave replication, automatic failover with Sentinel, consensus algorithms, and both client‑side and server‑side sharding to achieve high performance and high availability.
This article presents a detailed case study of moving a 2‑petabyte smart‑product data workload from Elasticsearch to MongoDB, covering business background, source cluster architecture, MongoDB design and resource planning, step‑by‑step performance optimizations, latency and cost comparisons, and guidance on when each database is appropriate.
The article explains how to evolve a simple single‑instance database into a highly available, horizontally sharded system, detailing offline scaling, online dual‑write migration, and a smooth four‑node expansion process with configuration changes, data migration, and cleanup steps.
This article explains how to handle multi-table cross-database transactions by aligning sharding strategies for related tables, using a case study of shop_order_detail and shop_order_stat, and provides detailed formulas and TDDL configuration code to ensure consistent routing within the same database shard.
This article provides a comprehensive analysis of Redis Cluster, covering node and slot assignment, command execution, resharding, redirection, fault‑tolerance, gossip communication, scaling strategies, configuration limits, and practical code examples for building and operating a high‑availability sharded Redis deployment.
This article explains why MySQL uses B+‑tree indexes, describes the left‑most prefix rule for composite indexes, offers practical index‑design and optimization tips, outlines MyBatis first‑ and second‑level caching, details master‑slave replication, and introduces common sharding strategies and their implementation considerations.
This article explains the signs that indicate a need for database sharding, outlines performance bottlenecks, presents optimization techniques such as SQL tuning, table redesign, read‑write separation, and then details vertical and horizontal splitting strategies, their implementation across single or multiple databases, and the added complexities like cross‑database joins, distributed transactions, and ID generation.
This article examines core MySQL architecture principles—high availability, performance, consistency, and scalability—then compares four common deployment patterns, analyzes their trade‑offs, and presents practical consistency solutions for both master‑slave replication and cache synchronization, ending with personal insights on evolution and sharding.
The article outlines eBay’s practical scalability principles—including functional partitioning, horizontal sharding, avoiding distributed transactions, aggressive asynchronous decoupling, moving work to async pipelines, virtualization, and intelligent caching—demonstrating how large‑scale web systems can be designed for linear growth and high availability.
This article presents a comprehensive case study of Ctrip's unified accounting platform, detailing its evolution from legacy systems, architectural redesign, component decomposition, logging, sharding, asynchronous processing, database strategies, exception handling, and daily settlement mechanisms to achieve agility, decoupling, and reusability.
This comprehensive guide explains MongoDB cluster components, common use cases, monitoring commands, basic operational tasks, data migration procedures, troubleshooting of production issues, and optimization recommendations to achieve high performance and scalability.
This article explains how Vivo designed a company‑wide comment middle‑platform, chose MySQL and MongoDB, deep‑dived into MongoDB cluster architecture, shard key strategies, and practical solutions for scaling, migration, and high availability in a high‑traffic environment.
This article presents practical best‑practice guidelines for configuring Elasticsearch in production, covering heap memory sizing, CPU considerations, shard and replica planning, hot‑warm node architecture, node role settings, common pitfalls, monitoring APIs, and troubleshooting tips.
This article provides a comprehensive technical guide to distributed transactions, covering ACID fundamentals, consistency models, sharding techniques, CAP and BASE theories, and detailed implementations of two‑phase, three‑phase, and TCC protocols, while highlighting their advantages, limitations, and practical considerations.
This article explains the fundamentals of transparent sharding middleware in NewSQL systems, covering data partitioning strategies, read/write separation, the complete SQL processing pipeline, protocol adaptation for MySQL/PostgreSQL, distributed transaction models such as XA, Saga and TCC, and essential database governance and online scaling techniques.
To support Gaode Taxi’s soaring order volume, the team expanded from four to eight ECS instances, re‑sharded 256 tables into 4,096, built a custom binlog‑to‑Kafka sync middleware for full‑load and incremental migration, implemented rigorous validation and repair processes, and employed a gray‑rollout with ABC verification, completing the migration without code changes or incidents.
This article details Vivo's journey in building a company‑wide comment platform, explaining why MongoDB was chosen over MySQL, how its sharded cluster is structured, the challenges of shard‑key selection, and the practical steps taken to scale and maintain the system.
This article details the end‑to‑end process of expanding a high‑traffic ride‑hailing order system on Alibaba Cloud, covering capacity planning, database sharding from 256 to 4096 tables, custom data‑sync middleware, validation, repair, and a gray‑release migration that achieved zero incidents.
The article describes Uber’s Gairos platform—a unified real‑time data ingestion, storage, and query system built on Kafka, Elasticsearch and RT‑Gairos—detailing its architecture, Uber use cases such as dynamic pricing, the scalability and reliability challenges faced, and the suite of optimization techniques (sharding, query routing, caching, index merging, template tuning, and data‑pruning) applied to achieve low latency and high throughput.
This article objectively compares NewSQL distributed databases with traditional middleware‑plus‑sharding solutions, examining architecture, distributed transactions, CAP constraints, high‑availability, scaling, storage engines, maturity, and practical adoption scenarios to help readers decide which approach best fits their needs.
This article objectively compares NewSQL distributed databases with traditional middleware‑plus‑sharding solutions, examining their architectural differences, distributed transaction handling, high‑availability mechanisms, scaling and sharding strategies, SQL support, storage engines, and maturity to help practitioners decide which approach best fits their workload and operational constraints.
This article explains InnoDB's record storage, primary and secondary index mechanics, composite index rules, and provides practical e‑commerce sharding and table‑design guidelines with real‑world case studies to help developers optimize MySQL performance and storage efficiency.
This article systematically refutes ten widespread misconceptions about MongoDB—covering scalability, versioning, data format, transactions, relationships, sharding, security, data loss, and suitability—while explaining its architecture, BSON storage, and best‑practice usage with illustrative code examples.
This article explains common I/O and CPU bottlenecks in databases, introduces horizontal and vertical sharding techniques, discusses tools, global primary‑key generation methods, Snowflake ID algorithm, migration challenges, and provides practical guidance on when and how to apply database partitioning to improve performance and scalability.
This article objectively compares NewSQL distributed databases with middleware‑based sharding solutions, examining their architectural differences, distributed transaction handling, high‑availability mechanisms, scaling and sharding strategies, SQL support, storage engines, and overall maturity to help readers decide which approach best fits their workload.
This tutorial walks through setting up a Redis Cluster on a single server with six nodes, covering node startup, handshake, slot assignment, master‑slave replication, command routing, failover handling, and practical scaling operations such as adding, rebalancing, and removing nodes.
