Practical Application of Kyuubi in Xiaomi’s Big Data Platform
This article details how Xiaomi integrated the open‑source Kyuubi SQL gateway into its evolving big‑data platform, describing the challenges of multiple SQL services, the architectural redesign for a unified, high‑availability service, performance gains, new features such as engine pooling and Z‑ordering, and future roadmap plans.
In response to fragmented SQL services, multiple authentication systems, and operational inefficiencies, Xiaomi built a unified big‑data development platform and identified Kyuubi as the optimal solution for a consistent, high‑availability SQL entry point.
The migration replaced legacy SQL Proxy with Kyuubi, leveraging its compatibility with Hive Thrift, resource isolation, and extensibility, while also integrating Trino and Doris via JDBC adapters.
Key architectural upgrades include a decoupled Kyuubi Server, Engine Manager, and containerized deployment on Kubernetes, enabling automatic scaling, graceful upgrades, and robust monitoring.
Performance improvements were demonstrated through TPC‑DS benchmarks and real‑world workloads, achieving up to 75% latency reduction and a ten‑fold increase in daily SQL processing volume.
New capabilities such as engine pooling, incremental result fetching, Z‑ordering, PlanOnly mode, and Scala code execution further enhance usability, resource efficiency, and support for diverse workloads.
Future plans involve automated multi‑engine routing, asynchronous HTTP‑based ETL submission, and continued expansion of Kyuubi’s ecosystem as a benchmark SQL gateway for big‑data scenarios.
DataFunSummit
Official account of the DataFun community, dedicated to sharing big data and AI industry summit news and speaker talks, with regular downloadable resource packs.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.