Shopee Druid Cloud Native Architecture Evolution: Design and Implementation

This article discusses the evolution of Shopee's Druid database from a physical machine architecture to a cloud-native architecture. It begins by introducing Druid as a high-performance, real-time analytical database with features like columnar storage, bitmap indexing, and SIMD acceleration. The article then outlines the problems encountered with the physical machine architecture, including stability issues (such as traffic spikes and slow queries), efficiency problems (like manual scaling and lack of priority management), cost issues (resource waste and high maintenance costs), and security vulnerabilities.

The cloud-native architecture addresses these issues by providing high stability through independent clusters and automatic recovery, efficiency through automatic scaling and traffic management, low cost through better resource utilization and reduced maintenance, and enhanced security through authentication and container isolation. The article details the architecture design, including six layers: business, platform, visualization, engine, GitOps, and Kubernetes. It explains the internal component interactions, monitoring systems, and the benefits of using Kubernetes for orchestration.

The implementation section describes how Shopee provides Druid on Kubernetes clusters for core businesses, along with Grafana for monitoring and various visualization tools like Turnilo, Superset, and Kibana. The article concludes with a summary of the benefits achieved through the cloud-native evolution and an appendix explaining basic concepts related to Druid, Kubernetes, and cloud-native technologies.