BestHub
Sign in
Tag

storage architecture

1 views collected around this technical thread.

Xiaokun's Architecture Exploration Notes
Xiaokun's Architecture Exploration Notes
Jun 8, 2025 · Databases

Why Data Replication Matters: Architectures, Formats, and Consistency Models

This article explores the principles of data replication, comparing shared memory, shared disk, and non‑shared storage architectures, detailing replication formats, consistency challenges, and various replication strategies such as synchronous, asynchronous, semi‑synchronous, and majority‑based approaches, helping engineers choose the right trade‑offs.

asynchronous replicationconsistencydata replication
0 likes · 12 min read
Why Data Replication Matters: Architectures, Formats, and Consistency Models
High Availability Architecture
High Availability Architecture
Jan 17, 2025 · Databases

Design and Implementation of a Multi‑Level Comment Storage System for Bilibili

This article presents a comprehensive design of Bilibili's comment service architecture, detailing the transition from TiDB to a multi‑level storage system based on Taishan KV, the data models, consistency mechanisms, retry and versioning strategies, and a hedging‑based degradation policy to ensure high availability under heavy traffic.

Taishan KVTiDBcomment system
0 likes · 13 min read
Design and Implementation of a Multi‑Level Comment Storage System for Bilibili
Bilibili Tech
Bilibili Tech
Dec 13, 2024 · Databases

Design and Implementation of a Multi-Level Storage Architecture for Bilibili Comment Service

The paper proposes a multi‑level storage architecture for Bilibili’s comment service that replaces TiDB with a custom KV store (Taishan) and Redis caching, introduces unstructured indexes, CAS‑based consistency, real‑time and offline reconciliation, and a hedged degradation strategy to boost reliability, read throughput, and scalability during traffic spikes.

NoSQLRedisTiDB
0 likes · 13 min read
Design and Implementation of a Multi-Level Storage Architecture for Bilibili Comment Service
Baidu Geek Talk
Baidu Geek Talk
Nov 6, 2024 · Cloud Computing

Baidu Canghai Storage Unified Technology Base: Architecture and Evolution of Metadata, Namespace, and Data Layers

Baidu’s Canghai Storage unifies metadata, hierarchical namespace, and data layers into a Meta‑Aware, three‑generation architecture that scales to trillions of metadata items and zettabyte‑scale data, using a distributed transactional KV store, single‑machine‑distributed namespace, and online erasure‑coding micro‑services to deliver high performance, low cost, and seamless scalability.

Erasure CodingNewSQLbig data
0 likes · 18 min read
Baidu Canghai Storage Unified Technology Base: Architecture and Evolution of Metadata, Namespace, and Data Layers
DaTaobao Tech
DaTaobao Tech
Aug 9, 2024 · Cloud Native

Deep Dive into RocketMQ: Storage Architecture and Consumer Processing Flow

RocketMQ stores messages in a sequential CommitLog while using a ConsumerQueue index for fast random access, and employs two looping threads—RebalanceService for load‑balancing queue assignments and PullMessageService for continuously pulling messages and dispatching them to user callbacks via a thread‑pooled consumer pipeline.

ConsumerJavaLoad Balancing
0 likes · 12 min read
Deep Dive into RocketMQ: Storage Architecture and Consumer Processing Flow
Architect
Architect
Aug 3, 2024 · Big Data

Kafka Storage Architecture Design: Deep Analysis and Implementation

This article thoroughly examines Kafka's storage architecture, analyzing its design motivations, storage mechanisms, log formats, partitioning, indexing, cleanup strategies, and performance optimizations such as sequential log writes, sparse indexing, page cache, and zero‑copy, providing insights applicable to other storage systems.

KafkaLog SegmentsSparse Index
0 likes · 22 min read
Kafka Storage Architecture Design: Deep Analysis and Implementation
Sanyou's Java Diary
Sanyou's Java Diary
Feb 1, 2024 · Backend Development

Message Queues Unveiled: From Decoupling to Platformization and Core Architectures

This article traces the two‑decade evolution of message queues—from early decoupling solutions like ActiveMQ, through high‑throughput designs such as Kafka, to modern platformized systems like RocketMQ and Pulsar—while explaining fundamental concepts, partitioning, and storage architectures that underpin today’s distributed messaging platforms.

KafkaMessage QueuePulsar
0 likes · 17 min read
Message Queues Unveiled: From Decoupling to Platformization and Core Architectures
Baidu Geek Talk
Baidu Geek Talk
Jan 31, 2024 · Cloud Computing

Cloud Computing in the AI-Native Era: Baidu Cloud's AI-Native Architecture and Latest Product Innovations

In his 2023 keynote, Baidu Vice President Xie Guangjun unveiled an AI‑native cloud architecture featuring 7th‑gen G7 servers, Kunlun R300 and Ascend 910B instances, a gateway with X86, programmable switches and FPGA, unified storage (TafDB, Aries, BOS, CDS, PFS), Baige 3.0 AI infrastructure, an intelligent computing network, GaiaDB 4.0, BMR Spark 3.2, SugarBot natural‑language analytics, distributed edge and private cloud, and video and low‑code platforms.

AI infrastructureBaidu CloudCloud Computing
0 likes · 19 min read
Cloud Computing in the AI-Native Era: Baidu Cloud's AI-Native Architecture and Latest Product Innovations
DataFunTalk
DataFunTalk
Aug 30, 2023 · Big Data

Design and Implementation of Baidu Cloud Block Storage EC System for Large‑Scale Data

This article presents Baidu Cloud's block storage architecture, comparing replication and erasure‑coding fault‑tolerance methods, detailing the challenges of applying EC to mutable block data, and describing a two‑layer append‑engine solution with selective 3‑replica caching, cost‑benefit compaction, and performance optimizations for low‑cost, high‑throughput storage.

Append EngineCompactionErasure Coding
0 likes · 14 min read
Design and Implementation of Baidu Cloud Block Storage EC System for Large‑Scale Data
Architects Research Society
Architects Research Society
Jun 10, 2023 · Big Data

Designing and Planning a Data Lake on Azure Data Lake Storage Gen2

This article provides a comprehensive guide to planning, structuring, securing, and managing a data lake on Azure Data Lake Storage Gen2, covering zone architecture, folder hierarchy, access control, file formats, scalability considerations, and best‑practice recommendations for big‑data workloads.

ADLS Gen2AzureData Lake
0 likes · 21 min read
Designing and Planning a Data Lake on Azure Data Lake Storage Gen2
DataFunTalk
DataFunTalk
May 21, 2023 · Databases

Graph Database Storage Techniques and Practices with Galaxybase

This article introduces RDF and property graph models, explains the core goals of graph database storage, compares mainstream storage solutions such as array, linked‑list and LSM‑Tree approaches, and presents practical deployment experiences of the Galaxybase distributed graph database.

GalaxybaseProperty GraphRDF
0 likes · 23 min read
Graph Database Storage Techniques and Practices with Galaxybase
DataFunTalk
DataFunTalk
Jan 14, 2023 · Databases

Evolution and Architecture of Graph Databases: From Early Designs to Modern Distributed Systems

This article surveys the development of graph databases, describing their underlying data models, storage designs across relational, native, document, and wide‑column systems, and reviewing representative modern distributed graph databases while discussing current challenges and future directions such as GQL standardization and graph‑AI integration.

NebulaGraphNeo4jNoSQL
0 likes · 29 min read
Evolution and Architecture of Graph Databases: From Early Designs to Modern Distributed Systems
Architecture & Thinking
Architecture & Thinking
Nov 23, 2022 · Backend Development

How RocketMQ Stores Messages: Inside the Broker’s CommitLog, ConsumeQueue, and IndexFile

This article explains RocketMQ's broker storage architecture, detailing the roles of CommitLog, ConsumeQueue, and IndexFile, and describes how messages are formatted, indexed, and written to disk using memory‑mapped files to achieve high throughput and reliability.

CommitLogConsumeQueueIndexFile
0 likes · 8 min read
How RocketMQ Stores Messages: Inside the Broker’s CommitLog, ConsumeQueue, and IndexFile
Architects' Tech Alliance
Architects' Tech Alliance
Oct 15, 2022 · Fundamentals

Key Technologies of Enterprise Flash Storage Systems: Global Load Balancing, Multi‑Stream Partitioning, and End‑to‑End I/O Prioritization

The article explains core flash storage techniques—including global load balancing, multi‑stream partitioning, and end‑to‑end I/O priority—by describing Active‑Active architectures, hash‑based request distribution, cache mirroring, data classification, and latency management to improve performance, reliability, and SSD lifespan.

Flash StorageI/O PriorityLoad Balancing
0 likes · 9 min read
Key Technologies of Enterprise Flash Storage Systems: Global Load Balancing, Multi‑Stream Partitioning, and End‑to‑End I/O Prioritization
Architects' Tech Alliance
Architects' Tech Alliance
Oct 10, 2022 · Fundamentals

All‑Flash Storage System Architecture and Key Functions (Dorado Flash Product Example)

The article explains the fully interconnected architecture of an all‑flash storage system, covering redundant FRU modules, RDMA‑based high‑speed networking, intelligent disk enclosures, SSD structure, wear‑leveling, bad‑block management, data redundancy, and the differences between SAS and NVMe protocols.

Data RedundancyNVMeRDMA
0 likes · 12 min read
All‑Flash Storage System Architecture and Key Functions (Dorado Flash Product Example)
DataFunTalk
DataFunTalk
Aug 9, 2022 · Databases

Graph Database Storage Technologies and Practices: Concepts, Core Goals, Technical Solutions, and Galaxybase Case Study

This article introduces graph database fundamentals, explains why graph databases are needed, outlines core storage goals such as index‑free adjacency, compares array, linked‑list and LSM‑tree storage schemes, and presents the design, performance advantages, and real‑world applications of the Galaxybase distributed graph database.

GalaxybaseLSM Treebig data
0 likes · 20 min read
Graph Database Storage Technologies and Practices: Concepts, Core Goals, Technical Solutions, and Galaxybase Case Study
DataFunSummit
DataFunSummit
May 19, 2022 · Databases

Designing a One‑Stop IoT Storage Solution: Architecture, Cost Optimization, and Performance

The talk outlines IoT data classifications, requirements, and proposes a one‑stop storage product using multi‑model support, columnar formats, compute‑storage separation, tiered storage, and query optimization to achieve ten‑fold cost reduction and ten‑fold performance gains.

Database DesignIoTcloud native
0 likes · 20 min read
Designing a One‑Stop IoT Storage Solution: Architecture, Cost Optimization, and Performance
vivo Internet Technology
vivo Internet Technology
Apr 20, 2022 · Backend Development

FastDFS Overview: Principles, Architecture, Upload/Download Process, Synchronization, and Storage Management

FastDFS is a lightweight, open‑source distributed file system written in C that uses a three‑component architecture—client, tracker server for load‑balancing and discovery, and storage servers with push‑based binlog replication—to handle high‑concurrency upload/download of small to medium files, support group‑wide synchronization, optional trunk storage, Nginx anti‑leech integration, and extensible deduplication via FastDHT.

Nginx ModuleSynchronizationUpload Download
0 likes · 15 min read
FastDFS Overview: Principles, Architecture, Upload/Download Process, Synchronization, and Storage Management
IT Architects Alliance
IT Architects Alliance
Sep 21, 2021 · Fundamentals

Analyzing Android 6.0 Storage Architecture and Communication Design

This article examines Android 6.0’s storage architecture, detailing the roles and interactions of MountService, Vold, and the kernel, illustrating module and process structures, communication flows via sockets, and highlighting performance considerations such as thread blocking and ANR issues.

AndroidMountServiceVold
0 likes · 14 min read
Analyzing Android 6.0 Storage Architecture and Communication Design