This article compares the data modeling and indexing techniques of three mainstream OLAP engines—Kylin, Druid, and ClickHouse—explaining their conversion processes, storage formats, query execution paths, and summarizing their strengths and limitations for aggregation and detail query scenarios.
The article explains how to evaluate and improve data warehouse design by measuring completeness, reusability and standardization, proposes concrete metrics such as cross‑layer reference rate and model reuse coefficient, and outlines a step‑by‑step process—from ODS control to domain division, dimension unification, fact‑table integration, model development and migration—while introducing the EasyDesign tool for systematic management.
This article shares the practical experience and thinking behind Ctrip's vacation data governance project, covering team efficiency optimization, demand sorting, data domain definition, warehouse layering, unified dimension modeling, metric standardization, and the overall benefits of a centralized data governance framework.
Xianyu’s new quality‑item framework evaluates each listing through product, supply‑demand, price, and seller attributes to compute a quality score, optimizes a balanced pool of high‑quality items, and—validated by back‑testing and simulations—boosts click‑through and payment conversion rates dramatically.
This article describes the background, metadata concepts, SPU and category‑attribute modeling, MySQL tree‑storage techniques, dynamic business configuration, rule‑engine and workflow‑engine solutions adopted in Tripyun‑Ctrip Cloud to build a reusable, scalable middle‑platform for the tourism industry.
The article describes how a lightweight big‑data analysis platform called MVP was built from scratch—using a User‑Event‑Config model, HDFS + ClickHouse + Spark, and four modules for metric monitoring, root‑cause alerts, deep growth analysis, and A/B testing—enabling real‑time insights in seconds instead of days and dramatically accelerating product‑growth operations.
This article presents a comprehensive design and implementation guide for a financial data warehouse, covering background needs, modeling methodology choices, a layered architecture, data quality monitoring, metadata management, naming and coding standards, and future development directions.
The article offers a detailed technical overview of OpenTSDB’s architecture and data model, explaining how it leverages HBase for scalable time‑series storage, describing core concepts, table schemas, ingestion flow, performance considerations, and future alternatives for large‑scale monitoring workloads.
This article explains the limitations of traditional REST APIs, introduces GraphQL as a flexible alternative, and details its core concepts, type system, server‑client architectures, deployment options, and popular tools and services for modern backend development.
This article compiles a comprehensive set of 32 interview questions covering data warehouse fundamentals, data platform construction, modeling approaches, real‑time architectures, data quality, governance, Hive optimization, and related analytical techniques to help candidates prepare for data engineering roles.
This article explains the concepts of data modeling and architecture, including logical data, process, and rule modeling, various data model types, master data management principles, metadata categories, and data quality management practices, highlighting their roles in enterprise information systems.
Lin Yuansheng, JD’s first post‑doctoral researcher, leverages his system‑engineering background and big‑data analytics to build models that identify black‑market accounts, improve network risk perception, and demonstrate how academic research can be rapidly applied to real‑world e‑commerce security challenges.
Facing growing order volumes and strict timeliness demands, Cainiao’s tech team overhauled its real‑time data warehouse by redesigning data models, adopting Flink for streaming computation, upgrading data services, and exploring innovative tools, sharing practical lessons and future directions for large‑scale logistics analytics.
This article examines how Ant Financial redesigned its payment system’s data layer after retiring mainframes, detailing RTO/RPO goals, CAP trade‑offs, vertical and horizontal sharding, blacklist‑based accounting DR, failover mechanisms for transactions, and the role of OceanBase in achieving strong consistency and low‑latency recovery.
The article explores a concern‑driven approach to data architecture, introducing a full‑scale schema modeling framework that balances data control, flexibility, push/pull delivery, and governance, and discusses multi‑level concerns, property‑graph standards, and practical scenarios for evolving schemas in complex business contexts.
This article explains how to design, implement, and maintain a robust Configuration Management Database (CMDB) by focusing on simple modeling, establishing data closure loops, and efficiently handling existing inventory, while leveraging Kafka, Flink, Elasticsearch, and Neo4j for fast querying and topology visualization.
This article describes AutoHome's data warehouse architecture, detailing its background, business pain points, layered model design (RDM, ADM, GDM, FDM, Stage/BDM, DIM, TMP), advantages in performance, cost, efficiency, quality, and various application scenarios including BI, analytics, and decision support.
The article explains how to formally name data schemas, data structures, and data models, clarifies their relationships, introduces physical, business, logical, and deployment schemas, and proposes strategic and tactical schemas to improve data architecture and management across organizations.
This article shares Cainiao's practical experience in constructing a real-time data warehouse, covering the shortcomings of the previous architecture, the evolution of data models, the migration to Flink with advanced features like retraction and timer services, and the modernization of data services and tooling to support high‑throughput logistics scenarios.
This article details how JD's MPM visual H5 page builder tackles complex data‑request challenges by introducing a unified data model, request aggregation, and three‑end rendering support, offering a scalable solution that improves performance, maintainability, and developer experience across web, app, and server‑side rendering.
This article explains the concept of granularity in data warehouse design, describing data models composed of structures, operations, and constraints, illustrating how granularity affects storage detail, query performance, and resource consumption, and recommending a dual‑granularity approach to balance efficiency and analytical depth.
This article provides a comprehensive overview of HBase, covering its LSM‑Tree based storage engine, key‑value data model, column‑family storage design, indexing mechanisms, major advantages and drawbacks, and typical scenarios where HBase excels for massive, high‑throughput data workloads.
The talk details Cainiao’s evolution of its real‑time data warehouse architecture, covering the original 2016 model, compute and service challenges, the 2017 multi‑layer data model redesign, migration to Flink, practical cases of state retraction, timeout statistics, smart optimizations, and the unified data service platform.
This article systematically describes the challenges, design principles, modeling methods, layered architecture, implementation steps, and standards used in building a comprehensive user behavior data warehouse for 58.com, highlighting practical experiences and future improvement directions.
This article explains the importance of good database design, walks through the stages of building an ER conceptual model, shows how to map entities, attributes, and relationships to relational schemas, and demonstrates how normalization (1NF, 2NF, 3NF, BCNF) improves schema quality using practical examples.
The article surveys modern big‑data architecture, contrasting Lambda and Kappa models, highlights common governance and integration pain points, and proposes an end‑to‑end platform featuring unified metadata, stream‑batch processing, one‑click ingestion, standardized modeling, intelligent query abstraction, and a comprehensive development IDE.
The article provides a comprehensive overview of the Entity‑Relationship (ER) model, covering its definition, conceptual, logical, and physical data models, notation variants, cardinality constraints, common pitfalls, semantic extensions, and limitations in relational database design.
This article provides a comprehensive overview of ClickHouse, covering its typical application scenarios, inherent limitations, common SQL syntax, default values, data types, materialized and expression columns, and detailed explanations of its various storage engines such as TinyLog, Log, Memory, Merge, Distributed, Null, Buffer, Set, MergeTree, ReplacingMergeTree, SummingMergeTree, AggregatingMergeTree, and CollapsingMergeTree, accompanied by practical code examples.
This article presents the OneData methodology for building a robust big‑data platform, detailing background challenges, goals, unified input and output strategies, model design, naming conventions, data‑cleaning rules, and the resulting business benefits and future outlook.
This guide explains InfluxDB’s schemaless data types, best practices for designing tag‑and‑field schemas, and how to choose and configure its in‑memory or TSI indexes, including key parameters such as max‑series‑per‑database and max‑values‑per‑tag for optimal performance.
This article explains the fundamentals of time‑series databases, introduces InfluxDB’s key components such as databases, measurements, retention policies, tags, fields, series, and points, describes the line protocol syntax, compares it with traditional databases, and outlines the design trade‑offs that shape its performance and limitations.
By adapting Alibaba’s OneData methodology, the project establishes a unified data‑warehouse architecture, standards, and governance framework—including consolidated business intake, standardized design layers, naming conventions, and delivery metrics—that resolves data‑quality issues, enhances scalability and reusability, and delivers faster, reliable data support for evolving business needs.
This article explains HBase's data model and key components, details column descriptor options such as BloomFilter, Compression, Versions, TTL, and MinVersion, and provides practical design guidelines for columns, rowkeys, high vs. wide tables, region pre‑splitting, and hotspot mitigation to achieve optimal performance.
This article details how Alibaba engineers built a digital mall by defining a unified device data dictionary, deploying Niagara IoT, implementing edge‑computing with IBOS, constructing a layered data‑center architecture, and delivering analytics, visualization, and service platforms for traffic, transaction, and membership insights.
This article details Mafengwo’s journey in constructing a data warehouse and data middle platform, covering the core three‑layer architecture, hybrid modeling approaches, the supporting toolchain for data synchronization, scheduling, and metadata management, and the design of an indicator platform for business analytics.
This article provides a comprehensive overview of Apache Cassandra, covering its origins at Facebook, key architectural features such as distributed design, elastic scalability, tunable consistency, and high availability, as well as typical deployment scenarios and notable companies that use it.
This article explains how to build a data warehouse from scratch, covering its definition, system and collaboration layers, ETL requirements, data layering design, modeling steps, common challenges, and governance practices such as temporary table management and coding standards.
This article explains how to model and implement a basic instant‑messaging system using Alibaba Tablestore's Timeline model, covering message storage, relationship management, metadata handling, real‑time notification, multi‑device synchronization, and provides complete Java code examples for each functional module.
Elasticsearch performance at billions‑scale can be dramatically improved by leveraging the OS filesystem cache, limiting indexed fields, separating hot and cold data, pre‑warming caches, and using scroll or search_after for pagination, while avoiding costly joins and ensuring the dataset fits in memory.
This article provides a comprehensive overview of Apache Parquet, covering its purpose, architectural components, nested data model, file structure, practical Hive commands for creating and inspecting Parquet tables, and a brief introduction to the TPC‑DS benchmark for performance testing.
To boost Elasticsearch query speed on billions of records, allocate sufficient filesystem cache memory, store only searchable fields, separate hot and cold data, warm up cache, avoid complex joins, and replace deep pagination with Scroll API or search_after for millisecond‑level responses.
This article explains why Elasticsearch can be slow on massive datasets, then details practical techniques—leveraging filesystem cache, pre‑heating hot data, separating hot and cold indices, designing lean document models, and avoiding deep pagination—to achieve sub‑second query performance at billions‑scale.
This guide outlines comprehensive MySQL naming conventions, table and column design rules, indexing best practices, SQL coding standards, and operational recommendations to improve performance, maintainability, and security while avoiding common pitfalls such as redundant indexes, unsafe data types, and excessive batch operations.
This article explains HBase as a high‑performance, column‑oriented distributed storage system, outlines its advantages and limitations, presents real‑world scenarios such as seller operation logs and message logs, and details its data structures, architecture components, and design considerations for big‑data applications.
This comprehensive guide explains Neo4j's label‑property graph model, node and relationship creation, Cypher syntax, indexing, constraints, schema inspection, and best practices for avoiding duplicate data, providing practical examples and performance tips.
This article analyzes a ride‑hailing business case, using SPSS to fit discount‑order and discount‑revenue curves, derives optimal discount points (84% for fastest order growth and 86% for maximum revenue), and discusses how these insights guide subsidy and marketing strategies.
This article explains Alibaba’s end‑to‑end approach to constructing an e‑commerce knowledge graph—detailing the background, challenges, data‑structuring methods, schema design, modular architecture, and deployment pipeline that enable deep user‑intent understanding across complex shopping scenarios.
This article explains what native multi‑model databases are, why they are needed, how they combine document, key‑value and graph models, and demonstrates their advantages through an aircraft maintenance management case study with concrete AQL queries.
This article outlines practical guidelines for designing Elasticsearch indices, comparing single versus time‑based indexes, detailing mapping settings, shard allocation strategies, and deduplication methods, while providing concrete examples and code snippets for effective search infrastructure management.
This article explains how to design a flexible product model for e‑commerce by evolving from simple category‑article structures to a richer schema that includes brands, attributes, specifications, and separate SKU entities, enabling effective filtering, search, and integration with other services.
This article presents a detailed case study of Baidu Waimai's traffic analysis platform, outlining the data challenges of high dimensionality and volume, the evaluation of OLAP engines, the adoption of Apache Kylin for pre‑computation, the end‑to‑end data modeling, cube construction, incremental builds, and integration with Saiku‑Mondrian reporting, while sharing practical lessons and performance gains.
The article explores a programmer’s view of software as code and data, distinguishes persistent and transient data, critiques JavaBean‑centric MVC architecture, and proposes key‑value Maps as a more flexible, decoupled alternative for backend development.
The article reflects on the nature of programs as code and data, distinguishes persistent and transient data, explains JavaEE MVC and multi‑layer architecture, critiques the overuse of JavaBeans, and advocates using key‑value Maps as a simpler, more flexible data‑transfer model.
This guide explains the fundamentals of conceptual data modeling with PowerDesigner, covering entities, attributes, identifiers, data items, and various relationship types, and shows step‑by‑step how to create and configure these elements in a CDM.
This article introduces Common Table Expressions (CTE) in SQL, explains how they improve query readability and enable writable temporary tables, demonstrates multi‑table modifications, advanced CRUD scenarios, implements a ring‑buffer using CTE, and discusses performance benefits and potential pitfalls.
The talk outlines MamaBang's evolution from MySQL to a multi‑zone MongoDB cluster, compares MongoDB with relational databases, discusses data‑model design, sharding, replication, transaction handling, operational challenges, and offers practical advice for teams considering MongoDB adoption.
This article details how Discord scaled from tens of millions to over a hundred million daily messages by replacing MongoDB with Cassandra, covering the motivations, data model design, handling of tombstones, performance results, unexpected issues, and future scalability plans.
This article reviews the latest Elasticsearch 5.x enhancements—including append‑only indexing, range fields, removal of the _all field, unified highlighter, keyword normalizer, multi‑word synonyms, field collapsing, cancellable searches, partitioned term aggregations, cluster allocation explain, Java REST client updates, cross‑cluster search, batched reduce phases—and previews the major features expected in Elasticsearch 6.0 such as sparse doc values, index sorting, sequence numbers, seamless rolling upgrades, type removal, index‑template inheritance, load‑aware shard routing, and X‑Pack extensions like SQL and machine learning.
This article details a comprehensive e‑commerce data architecture, covering product module design, SKU handling, packaging strategies, order schema, promotion tracking, visit analytics, and reporting best practices, offering practical insights for building robust backend systems.
This article explains the concept of user profiling, outlines its four-stage construction process, discusses the significance of tagging users, and details practical data modeling techniques—including static and dynamic data sources, weight calculations, and real‑world examples—aimed at improving precision marketing and recommendation systems.
This article compares traditional relational table design with HBase's column‑family model, showing how tables, rows, column families, and versioned cells are defined and how data is inserted using HBase's put commands.
This article explains how to define data elements, choose meaningful names, apply ISO‑11179 naming patterns, select appropriate SQL data types, and use domains, constraints, and default values to build clear, maintainable, and high‑performance relational database schemas.
This article explores how an ideal web framework—featuring a unified data model, reusable code, event‑driven logic, and language‑agnostic architecture—can be realized in practice, discussing frontend and backend realities, graph‑based data modeling, and the benefits for maintainability and extensibility.
The article explains the goals of user behavior analysis, details two implementation methods for single-step and multi-step path conversions using log preprocessing, adjacency matrices, and KMP pattern matching, and illustrates these techniques with a Baidu Doctor product line case study.
The article compares relational databases like MySQL—highlighting their consistency, transaction support, and join capabilities—with NoSQL alternatives, explaining each type’s strengths and limitations in handling massive writes, schema changes, variable fields, and simple queries, and guides developers on when to choose each.
This article delivers a comprehensive guide to Greenplum deployment, covering early architecture planning, data‑model design, daily maintenance best practices, system‑table management, diagnostic tools like gpcheckcat, and detailed troubleshooting techniques for persistent tables and other common issues.
This article traces the evolution of database models—from early hierarchical and network structures through the relational revolution and entity‑relationship diagrams to modern distributed and multi‑tenant designs—highlighting key concepts, technical challenges, and the theoretical foundations that shaped today’s data storage systems.
This article explores how ideal web frameworks could evolve by separating data description from logic, using event‑driven architectures, graph databases, and cross‑language integration to achieve minimal, maintainable, and extensible code, while contrasting current practices like React, Angular, and Meteor.
The article explores how ideal web frameworks could evolve by separating data description from logic, using event‑driven architectures, graph databases, and cross‑language components to achieve minimal, maintainable, and extensible code, while discussing practical implementations, challenges, and philosophical foundations.
This article surveys major NoSQL data models—including document, graph, relational, object, key‑value, BigTable‑type, data‑structure, and grid databases—and provides practical guidance on selecting the most suitable database for various application requirements such as scalability, consistency, transaction support, and data complexity.
This article compares relational databases with distributed document-oriented NoSQL databases, explaining their scaling models, data modeling differences, and offering practical advice on when and how to adopt document databases such as SequoiaDB for flexible, high‑performance applications in the big‑data era.
This article explains how retailers can replace costly ETL‑driven vendor systems with a centralized MongoDB‑based service architecture, detailing data models for products, variants, store‑specific pricing, and multi‑facet search, and showing query and indexing techniques to achieve fast, reliable catalog access.
