Remote StateBackend for Flink: Design, Optimizations, and Cloud‑Native Migration

To enable Bilibili’s cloud‑native migration, the team built a RemoteStateBackend that moves Flink’s keyed state to the Taishan KV store, using deterministic KeyGroup placement, per‑shard snapshots, asynchronous write batching, off‑heap caching with Bloom‑filter filtering, and a fixed‑size memory model, which together reduce checkpoint overhead, improve disk utilization, and accelerate rescaling for more than one hundred production jobs.

CloudNativeFlinkPerformanceOptimization

0 likes · 18 min read

Remote StateBackend for Flink: Design, Optimizations, and Cloud‑Native Migration

Aikesheng Open Source Community

Sep 29, 2020 · Databases

Understanding Prometheus Local Storage (TSDB) and Its Architecture

This article explains Prometheus's built‑in time‑series database (TSDB), covering its concepts, storage configuration, block structure, write‑ahead log, mmap reads, inverted indexing, data compression, and remote storage integration for scalable monitoring.

MonitoringPrometheusRemoteStorage

0 likes · 8 min read

Understanding Prometheus Local Storage (TSDB) and Its Architecture

Big Data Technology & Architecture

Jun 24, 2020 · Operations

Design and Implementation of a General Business Monitoring and Alert Engine Using Prometheus and ClickHouse

This article describes how a company replaced its Zabbix‑based monitoring with a scalable, Prometheus‑driven alert engine that leverages ClickHouse for storage, remote‑storage integration via Prom2Click, and materialized views to provide flexible, SQL‑based business metric alerts.

ClickHousePrometheusRemoteStorage

0 likes · 11 min read

Design and Implementation of a General Business Monitoring and Alert Engine Using Prometheus and ClickHouse