0 views collected around this technical thread.
To replace Elasticsearch, the team built a ClickHouse‑based log platform that uses table‑level TTL and custom storage policies to automatically move data among hot (ESSD PL1), cold (ESSD PL0) and archive (OSS) disks, achieving faster writes, comparable query latency, and over 50 % cost reduction.
Kafka’s architecture consists of brokers forming clusters, producers publishing to topics split into partitions with replicas, consumers organized in groups pulling messages by offset, ZooKeeper managing metadata, and log‑based storage using append‑only files, indexes, and zero‑copy, while configurable acknowledgment, batching, and replication ensure high throughput and fault‑tolerant reliability.
This article describes how Bilibili redesigned its log service by replacing Elasticsearch with ClickHouse, introducing OpenTelemetry‑based logging, optimizing storage, query, and alerting components, and enhancing ClickHouse features such as configuration tuning, Map types, and implicit columns to achieve higher performance, lower cost, and better observability.
Bilibili’s Log Service 2.0 replaces its Elastic‑Stack pipeline with an OpenTelemetry‑driven architecture that writes logs via high‑performance Go/Java SDKs to ClickHouse, delivering ten‑fold write throughput, two‑fold query speed, one‑third storage cost, a custom query gateway, visualization UI, and advanced alerting.
This article provides a comprehensive technical overview of Apache Kafka, covering its distributed messaging architecture, key features such as high‑throughput read/write, replication, partitioning, consumer group mechanics, offset management, rebalance processes, and practical code examples for synchronous and asynchronous offset commits.
The article compares ClickHouse, Elasticsearch, and MySQL for log storage and analysis, highlighting ClickHouse's active development, SQL support, JSON handling, performance, storage efficiency, integration options, and practical query examples to help developers decide the best backend for observability pipelines.
This article analyzes the characteristics of log data, explains why HBase is chosen for log storage, discusses the shortcomings of self‑built indexes, and presents optimization strategies such as rowKey design, filter usage, coprocessor integration, and third‑party indexing to improve query performance.