From the 1962 IBM 1311—an appliance‑sized, 9‑pound disk pack holding just 1.5 MB—to today’s pocket‑sized terabyte USB drives, this article traces six decades of magnetic storage breakthroughs, cost drops, and the societal impact of ever‑shrinking data capacity.
This guide shows how to implement a PHP function for user registration that validates input, hashes passwords, stores user details in a MySQL database, and a separate function for generic data storage, while highlighting key security considerations.
The article explains why MySQL recommends storing IPv4 addresses as unsigned 32‑bit integers instead of strings, detailing space savings, faster range queries, and indexing benefits, while also noting readability drawbacks and providing MySQL functions and Java code for converting between string and integer representations.
JD Advertising built a massive ad‑data warehouse on Apache Doris, reaching nearly 1 PB and 18 trillion rows, then implemented a hot‑cold data tiering strategy—first a lake‑based approach, later a native tiering solution in Doris 2.0—reducing storage costs by 87% and boosting query performance over tenfold.
This tutorial walks through using Python to crawl Weibo, covering environment setup, three login methods, data extraction functions for user info, posts and comments, anti‑crawling strategies, storage to CSV or MySQL, a full example script, and legal considerations.
This article explains MySQL's storage architecture, why using DELETE to remove rows is discouraged, compares DELETE, DROP, and TRUNCATE in terms of speed, space reclamation, transaction behavior, and auto‑increment handling, and provides practical commands and optimization tips.
BiFang is a lake‑stream integrated storage engine that merges Apache Pulsar message‑queue capabilities with Iceberg data‑lake features, providing a single unified data store with full‑incremental queries, sub‑second visibility, exactly‑once semantics, and seamless integration with Flink, Spark, and StarRocks for both real‑time analytics and batch processing.
JD Advertising built a petabyte‑scale ad analytics service on Apache Doris, identified a hot‑cold access pattern, and implemented a native cold‑hot tiering solution (upgrading to Doris 2.0 and optimizing schema changes) that cut storage costs by ~87% and boosted concurrent query capacity over tenfold while simplifying operations.
The article explains that using a 32‑bit UNSIGNED INT to store IPv4 addresses in MySQL saves space and improves index and range‑query performance, outlines the storage savings compared to VARCHAR, mentions the need for manual conversion, and provides MySQL and Java code examples for converting between string and integer representations.
The article explains how generative AI agents combine language models, tool integration, self‑guided planning, prompt‑engineering frameworks, extensions, function calls, and vector‑based data storage to create adaptable, retrieval‑augmented systems that can interact with real‑world APIs and perform complex tasks.
This article surveys several lightweight Python databases—including PickleDB, TinyDB, ZODB, Durus, Buzhug, Gadfly, and PyTables—detailing their main features, typical use cases, cautions, and providing basic code examples to help developers choose and apply the right storage solution for small‑scale or prototype projects.
Using an unsigned INT to store IPv4 addresses in MySQL saves space and enables efficient range queries, while strings are larger and slower; the article explains these advantages, outlines conversion functions INET_ATON/INET_NTOA, shows equivalent handling for IPv6, and provides Java utilities for bidirectional conversion.
This tutorial explains how to implement user registration and data storage in web applications using PHP functions, covering input validation, password hashing, MySQL connection, SQL insertion, and providing complete code examples for both registering users and storing generic data.
This article explains Apache Hudi's clustering service, detailing its workflow, three execution modes, and layout optimization strategies—including linear, Z‑order, and Hilbert space‑filling curves—to improve storage locality and query performance in large‑scale data lake environments.
This article explains why SQLite, the lightweight embedded relational database, is the most widely deployed database in the world, detailing its origins, core features, and diverse usage scenarios across mobile, embedded, desktop, data analysis, and web acceleration contexts.
This article explains how to use PHP functions to create a user registration system with input validation, password hashing, and MySQL insertion, and also demonstrates a generic data‑storage function that connects to a database and saves arbitrary data, highlighting key security and implementation steps.
This article details the evolution of ByteDance's TLS (Tinder Log Service) from a Loki‑based prototype to an EB‑scale, cloud‑native log system, covering its core properties, data organization, architecture, caching, hybrid storage, private codec, ecosystem compatibility, intelligent features, and real‑world case studies.
This article explains the fundamentals of data storage in computers, covering the concepts of RAM and DRAM, the role of capacitors and transistors, and how memory controllers and CPU caches work together to manage and accelerate access to binary information.
This article demonstrates how to use PHP functions to implement user registration and data storage, covering input validation, password hashing, MySQL database connection, SQL insertion, and returning operation results, while highlighting security considerations and practical code examples.
This article shares Ant Group's extensive experience in big data governance, detailing the overall data governance framework, data quality management, data storage governance, and future considerations, illustrated with practical cases and strategies for ensuring compliance, reliability, and cost efficiency.
Didi replaced its Elasticsearch‑based log platform with ClickHouse, redesigning architecture into isolated Log and Trace clusters, using hourly‑partitioned MergeTree tables and aggregating views to handle petabyte‑scale writes, diverse low‑latency queries, and high QPS, achieving over 400 nodes, 40 GB/s throughput, 30 % cost savings and four‑fold query latency reduction.
Zarr provides a modern, chunked and compressed storage format that lets you treat massive NumPy arrays like in‑memory objects, offering on‑demand loading, flexible back‑ends (disk, S3, zip), automatic caching, resizing, parallel reads/writes, and superior performance compared to traditional mmap‑based memmap files.
This article explains the hierarchy of digital storage units—from the single bit up to speculative sizes like Brontobyte and beyond—detailing their values, typical examples, and real‑world analogies such as characters per byte and the amount of text each unit can hold.
This article explains the structure of NAND flash memory, how MOSFETs with floating‑gate and tunneling layers perform write operations for logical 0 and 1, how read operations detect stored electrons, and how matrix control enables block‑level access.
The article explains the four main categories of computer storage—primary, secondary, tertiary, and offline—detailing their connection to the CPU, typical devices, the concept of direct‑attached storage, and an overview of RAID technology for performance and redundancy.
Vector databases store and search large-scale vector data, and in 2023 the five leading solutions—Chroma, Pinecone, Weaviate, Milvus, and Faiss—offer scalable, high-performance options for applications such as LLM-driven services, audio search, recommendation systems, image/video analysis, and semantic retrieval across various industries.
The article argues that the era of big data is ending, showing that most organizations store only modest amounts of data, that storage costs outweigh benefits, and that modern cloud and analytics tools allow efficient processing without needing massive datasets.
The forum explores the rapid growth and complexity of knowledge graphs, addressing storage and computation challenges through expert talks on graph database storage, query languages, practical implementation, and large‑scale financial knowledge graph platforms, offering attendees deep technical insights and hands‑on guidance.
This article explains how Zhihu adopted HDFS erasure coding to reduce storage costs, outlines cold‑hot file tiering policies, describes the EC conversion workflow and the custom EC Worker tool, and details methods for detecting and repairing damaged EC files in a Hadoop environment.
The article explains the fundamentals of data storage in computers, describing how binary data is saved using memory modules, the differences between static SRAM and dynamic DRAM, and the role of memory controllers in translating addresses and managing refresh cycles.
The talk outlines Baidu’s Baige AI IaaS solution for autonomous driving, detailing a low‑cost, high‑efficiency cloud stack that accelerates data access fivefold, boosts model training speed up to 391 %, cuts inference latency by 90 %, reduces simulation costs by 60 %, and explains the underlying storage, compute, container and GPU virtualization technologies.
This article examines HBase’s high reliability and performance as a column‑oriented NoSQL store, outlines its advantages and limitations, presents two practical use cases from e‑commerce, and details its data model, architecture components, and design considerations for effective deployment.
This article provides a comprehensive overview of the big data ecosystem, detailing nine essential technology categories—including data collection, storage, computation, analysis, resource management, retrieval, underlying infrastructure, and cluster installation—while comparing popular tools and illustrating their typical use‑cases with diagrams.
The article explains that using a 32‑bit UNSIGNED INT to store IPv4 addresses in MySQL saves space and improves indexing and range queries, outlines the conversion functions and their performance, discusses readability drawbacks, and provides Java code for bidirectional conversion between string and integer representations.
The article explains why MySQL’s high‑performance guide recommends storing IPv4 addresses as a 32‑bit UNSIGNED INT instead of VARCHAR, detailing space savings, faster range queries, conversion functions (INET_ATON/INET_NTOA), associated drawbacks, and provides Java code to convert between string and integer representations.
The 5th Techo TVP Developer Summit explored the surge of data, the strategic role of NoSQL in digital transformation, presented cutting‑edge trends, performance breakthroughs, cloud‑native multi‑model solutions, and real‑world case studies from finance to gaming, highlighting future directions for database technology.
This article provides a comprehensive overview of Oracle's database products, explaining why they matter, detailing the physical and logical storage structures, instance processes and memory areas, and tracing the evolution from early versions through RAC, multitenancy, and the latest 18c features.
This article explains how Prometheus, a data‑driven monitoring system, leverages a time‑series database (TSDB) to handle massive metric volumes, perform efficient queries, and enable powerful calculations such as recording rules for pre‑computed results.
This article explains Elasticsearch’s distributed cluster design, covering core concepts such as nodes, indices, shards, and replicas, compares mixed and tiered deployment models, examines data‑layer storage options, and discusses two typical distributed system architectures with their trade‑offs.
This article explains the core components of database systems, including their client‑server architecture, query processing, storage engine modules, classification by storage media (memory vs. disk), and the differences between row‑oriented and column‑oriented data layouts, concluding with future topics to explore.
The article examines how to efficiently store read/unread status for group chat messages by replacing per‑user lists with compact bitmap structures, discusses handling member exits, presents C++‑style struct definitions, and quantifies storage savings compared to the naïve 8‑byte per‑user approach.
This article explains the core concepts of Elasticsearch—including nodes, indices, shards, and replicas—covers its cluster and data‑layer architectures, compares mixed and tiered deployment models, and discusses the advantages and drawbacks of replica‑based distributed storage systems.
This article explains the limitations of CSV for big datasets, introduces the Feather binary format, shows how to install and use it with Python and pandas, and compares its saving/loading speed and storage size against CSV, highlighting Feather's advantages for efficient data handling.
This article explains the architecture of Elasticsearch clusters, covering node roles, index, shard and replica concepts, deployment models, data storage mechanisms, and compares two distributed system designs—local‑file‑system and shared‑file‑system—highlighting their advantages and trade‑offs.
The article explains how Prometheus, a data‑driven monitoring system, handles massive time‑series data using its TSDB storage engine, detailing concepts, query examples, storage characteristics, indexing mechanisms, and the benefits of pre‑computing rules for efficient monitoring at scale.
This article reviews the evolution of next‑generation data storage technologies, covering advances in storage media such as all‑flash and non‑volatile memory, modern storage architectures like software‑defined and hyper‑converged systems, emerging protocols (NVMe, NVMe‑oF), cloud‑based application models, and intelligent operation approaches.
The article explains what a data lake is, its origins, key characteristics such as storing all raw data, flexible access, and low‑cost storage, compares it with traditional data warehouses, discusses advantages, common criticisms, and the types of users who can benefit from it.
The Mobile SkyNet platform adds a dedicated mobile monitoring layer to SaaS services, using Zanlogger for error, warning, and info logs, Kafka‑HBase pipelines for high‑throughput storage, WeChat‑based alerting, and an MPaaS console for issue visualization, reducing mobile‑side incidents by about twenty percent.
This article explains how Python persistence works by serializing objects with pickle and cPickle, compares file‑based and database storage, demonstrates basic and advanced usage—including handling circular references, custom classes, and versioning—and offers practical tips for maintaining pickled data across code changes.
This article explains Python object persistence, covering the concepts of serialization with pickle and cPickle, various storage mechanisms, handling of complex objects, reference cycles, class instance pickling, versioning strategies, and advanced techniques such as custom state methods and Pickler/Unpickler usage.
The article details JD's transition from a fragmented C‑end invoice service to a unified Invoice Center, describing the original user‑experience and architectural flaws, the DDD‑based layered redesign, performance‑boosting data and file‑handling strategies, achieved operational gains, and the planned next‑year initiatives.
This article provides a comprehensive overview of HDFS—including its design goals, core components, data read/write workflows, high‑availability mechanisms, federation, storage policies, colocation benefits, and practical usage scenarios—explaining why it is the foundational distributed file system for large‑scale data processing.
San CLI UI combines a San‑based client, a Node.js/Express GraphQL server, and lowdb file storage, enabling custom routing, component enhancements, and a versatile plugin system—supporting widgets, configurations, tasks, and custom views—managed through ClientAddonApi and PluginManager with IPC communication.
This article shares the Ctrip flight big‑data platform’s journey of evaluating and migrating data storage from Hive, MySQL and Redis to CrateDB, covering performance requirements, query patterns, maintenance challenges, containerization, and production results that reduced interface latency and resource consumption.
This article provides an in‑depth analysis of Apache Druid V1’s column‑oriented storage format, covering its dictionary, encoded dimension values, bitmap inverted index, array handling, and how these structures are used during query execution, illustrated with diagrams and code examples.
This article explains why ClickHouse, the high‑performance columnar OLAP database, can return results on billions of rows within seconds, detailing its columnar storage, MergeTree engine, and how JD Cloud deploys and optimizes it on Kubernetes for scalability and reliability.
This article explains the concepts, principles, and benefits of storage virtualization, covering host‑level, network‑level, and storage‑level approaches, and compares NAS and SAN technologies while highlighting how virtualization creates unified storage pools and reduces total cost of ownership.
This article provides a detailed introduction to HDFS, covering its application scenarios, core architecture, fault‑tolerance benefits, drawbacks such as high latency and small‑file inefficiency, essential shell and API commands, cluster management procedures, and newer Hadoop 2.0 features like HA, Federation, snapshots, ACLs, and heterogeneous storage.
This article explains the fundamentals of data storage, covering hard‑disk hardware, internal structures, logical volumes, file systems, and the differences between direct‑attached, network‑attached, and storage‑area network solutions.
This article provides an in‑depth analysis of Apache Druid V1's column‑oriented storage format, covering dimension structures, dictionaries, variable‑length integer encoding, inverted indexes, array handling, and how these components are used during query execution.
This article reviews the challenges of compressing massive time‑series data, surveys existing methods, and introduces a novel two‑stage deep reinforcement learning framework (AMMMO) that adaptively selects compression modes, demonstrating significant compression ratio improvements and high throughput on large‑scale IoT and server workloads.
This article explains how to build user profiles by distinguishing persona from profile, describing the evolution of ID‑mapping techniques, designing a multi‑layer tag system, implementing statistical, interest, and model tags, storing the data in Hive, HBase, Codis and Elasticsearch, and finally evaluating profile timeliness, coverage and accuracy.
This article examines WeChat's approach to Android data storage issues, detailing the limitations of internal and external storage, the gradual migration strategy, and the design of the wechat‑vfs component that provides abstracted file migration, encryption, and cleanup capabilities.
This article provides a comprehensive overview of HDFS, covering its purpose, architecture, read/write mechanisms, replication strategies, component responsibilities, common command‑line tools, and the advantages and disadvantages of using Hadoop Distributed File System for large‑scale data storage.
This article explains Elasticsearch's cluster architecture, including nodes, indices, shards, replicas, deployment models, and data layer storage, and compares two types of distributed data system designs—local file‑system based and shared‑storage based—highlighting their advantages and trade‑offs.
Learn how to start web scraping with Python by mastering the three core steps—fetching, analyzing, and storing data—using urllib and requests, handling login, evading anti‑scraping measures like user‑agents and IP proxies, and saving results to JSON, CSV, or MongoDB.
The article details NetEase Open Courses’ feed‑stream architecture, describing how content ingestion, multi‑level filtering, vertically and horizontally split storage, Elasticsearch indexing, two‑tier caching, and micro‑service orchestration combine to deliver personalized, high‑availability course feeds while addressing scalability, consistency, and operational challenges.
This article reviews a decade of MaxCompute development, covering its origins, core technologies, performance gains, ecosystem integration, intelligent features, competitive positioning, and commercialization, while highlighting the platform's role as Alibaba's central big‑data compute engine.
This article provides a concise overview of three data storage solutions—Hive, MongoDB, and Redis—detailing their core concepts, operational principles, typical use cases, and key characteristics to help developers choose the appropriate technology for various workloads.
This article analyzes the shortcomings of traditional relational databases in big‑data scenarios and introduces five major NoSQL categories—columnar, key‑value, document, full‑text search, and graph databases—detailing their principles, advantages, disadvantages, common implementations, and appropriate use cases to guide storage technology selection.
This article explains how high‑performance computing (HPC) supports oil‑field exploration by detailing the stages of seismic data acquisition, processing, and interpretation, the demanding computational and storage requirements, parallel communication patterns, checkpointing, and data lifecycle management, illustrating the role of HPC in modern geophysical workflows.
The article explains how cloud-driven demands and diverse data types have spurred the rise of multi-model databases, detailing their architecture, storage structures, compression techniques, and access methods using SequoiaDB as a concrete example.
Meitu’s Naix distributed bitmap system accelerates massive user‑data analytics by using a three‑layer architecture, sharded RoaringBitmap storage, and PalDB, delivering over 600× faster queries than Hive, supporting fast generation plugins, fault‑tolerant replication, and millisecond‑level RPC query responses while reducing storage by 67%.
Shingled Magnetic Recording (SMR) uses overlapping tracks to dramatically increase disk surface density, lowering cost per gigabyte, but it eliminates random writes, requiring new zone‑based management and exposing standards like ZBC and ZAC for host‑aware and drive‑managed implementations.
This article details the architecture and evolution of a headline recommendation system, covering data aggregation, storage strategies using MySQL and Redis, challenges with reload latency and memory usage, and the optimizations—including data separation, Redis migration, and query pipeline improvements—that enabled scalable, efficient backend operations.
This article explains the layered architecture of large‑scale web systems, covering flexible component choices, load distribution strategies, business service and communication layers, storage options from file to object systems, and key evaluation criteria such as cost, scalability, security, and maintainability.
Alluxio, formerly Tachyon, is a memory‑centric distributed file system that unifies heterogeneous big‑data storage backends, optimizes small files, and provides a fast, unified data access layer between storage systems like S3 or HDFS and compute frameworks such as Spark or Hadoop.
This article examines the core technologies of big data—including data collection, storage, management, analysis, and mining—highlighting architectural challenges, analysis techniques, storage solutions, ETL processes, and the interplay between big data and cloud computing, while emphasizing practical implementation considerations.
This article explains the layered architecture of a web system, covering flexible component choices, load‑balancing techniques, business service and communication layers, various storage options—including file, block, and object storage—and key evaluation criteria for building robust, cost‑effective solutions.
This article provides a comprehensive overview of HBase, covering its distributed architecture, component roles, data organization, read/write mechanisms, and best practices for schema and region design to ensure efficient big‑data storage and retrieval.
This article outlines the extensive technology stack behind Airbnb, covering its programming languages, client and server libraries, web server, data storage solutions, server services, and underlying hardware infrastructure that support its operations across 190 countries.
Databases originated in the 1960s‑1990s when the United States consolidated wartime intelligence into computer‑stored Data Bases, and a review of their development from 1962 to 2016 reveals a relentless stream of technological breakthroughs that continuously enrich everyday life.
