2021 Database Landscape: Cloud, HTAP, Open Source, and DBaaS Practices

Technical Changes: Cloud Technologies, HTAP Rise, Hadoop Decline

The biggest shift in data technology is the impact of cloud infrastructure on databases, with future innovations focusing on the intersection of distributed databases and cloud services. The underlying architecture of databases is changing, moving from hardware‑centric assumptions to API‑centric models like AWS S3.

Cloud influence is profound; for PingCAP, building TiDB Cloud now requires far more effort than developing the database kernel itself.

HTAP (Hybrid Transaction/Analytical Processing) has become mainstream, addressing the challenges of massive data volumes and fragmented data silos by merging OLTP and OLAP capabilities, simplifying tech stacks and enabling real‑time analytics.

PingCAP published a paper in 2020 titled "TiDB – A Raft‑based HTAP Database" that laid the theoretical foundation for HTAP, which has since become widely adopted.

Apache announced the retirement of many Hadoop projects to the Apache Attic, reflecting Hadoop's diminishing role as real‑time analytics and unified tech stacks gain prominence.

Ecosystem Changes: Open‑Source Databases as the Mainstream

2021 was the year open‑source entered the mainstream of enterprise software in China, even being mentioned in the national "14th Five‑Year Plan". Open‑source databases now surpass commercial ones in deployments, and companies like Databricks and MongoDB have seen valuations double.

PingCAP’s open‑source strategy has accelerated user growth and rapid code iteration, with TiDB 1.0 released in 2017 and now serving over 2,000 customers and 1,600 contributors, ranking sixth globally in CNCF contributions.

Academic‑industry collaboration has deepened, exemplified by SIGMOD’s first China venue and PingCAP’s co‑hosting of the VLDB Summer School, fostering talent and bridging research with practice.

DBaaS Practical Implementation

Cloud Reshaping Database Business Logic

Open‑source maturity and cloud standardization enable new business models beyond protecting core scenarios, focusing on optimizing developer experience, leveraging cloud cost efficiencies, and viral PLG growth.

Cost Savings: Decoupled Architecture Design

Cloud‑native technologies address cost by separating compute and storage, using services like AWS EBS GP3 to flexibly allocate resources, and anticipating further separation of network, memory, and CPU cache.

Security

In the cloud, security expands from internal RBAC to comprehensive network‑to‑storage controls, relying on cloud‑provided mechanisms such as key management and standardized pricing models.

Operational Automation

Scaling delivery requires automating operations; TiDB uses Kubernetes Operators, Gardener for multi‑cluster federation, and Pulumi scripts to codify all operational tasks.

Multi‑Region SLA and Compliance

Supporting global customers demands multi‑AZ, multi‑region deployments and compliance with regulations like GDPR, which cloud providers facilitate through native capabilities.

Future Outlook

Key areas of interest include truly cloud‑native distributed OLTP databases, AI‑driven indexing (e.g., Lean Index), and micro‑service‑based storage engines that separate compaction from serving using serverless and shared storage technologies.

TiDB launched a 12‑month free developer trial in November 2021, offering HTAP‑enabled, containerized deployments with dedicated block storage at tidbcloud.com .

Author: Huang Dongxu, Co‑founder & CTO of PingCAP, veteran software engineer with extensive experience in distributed systems and open‑source databases.