Building a Log Platform with Native Kibana and ClickHouse (CKibana)

During rapid business growth, the volume of log data increases dramatically, making traditional ELK and Elasticsearch‑based log systems costly, unstable, and performance‑constrained. Many companies (e.g., Ctrip, Bilibili, Cloudflare, Uber) have switched their log storage to ClickHouse, prompting our own migration from Elasticsearch to ClickHouse.

Since 2020, our primary log system has gradually replaced Elasticsearch with ClickHouse, achieving a 70% cost reduction (to 30% of the original) and stable handling of over 5 trillion logs per day during the National Day peak.

The key challenge after storage migration is preserving the familiar Kibana UI experience for users without requiring them to learn a new query language or UI.

Our solution adds a Proxy layer between native Kibana and Elasticsearch, converting Kibana queries into ClickHouse syntax and returning Elasticsearch‑compatible responses. This open‑source proxy, named CKibana , enables seamless use of Kibana dashboards on top of ClickHouse data.

CKibana acts as a proxy that translates chart requests to ClickHouse queries, simulates Elasticsearch responses, and forwards them to Kibana. It also retains Elasticsearch for caching and metadata storage.

To start CKibana you need JDK 17+ and run:

java -jar ckibana.jar

Then modify the Kibana configuration to point the Elasticsearch address to the CKibana endpoint (see screenshot).

The overall component stack consists of:

Kibana : UI for business users.

Elasticsearch : stores Kibana metadata and provides query caching.

ClickHouse : stores the raw log data.

CKibana : the proxy that converts queries and forwards results.

Configure ClickHouse connection via:

curl --location --request POST 'localhost:8080/config/updateCk?url=ckUrl&user=default&pass=default&defaultCkDatabase=ops'

Whitelist the indices that should be switched to ClickHouse:

curl --location --request POST 'localhost:8080/config/updateWhiteIndexList?list=index1,index2'

Field type mapping between Elasticsearch and ClickHouse:

es type

ck type

keyword

String

text

String

ip

String (auto‑detect IPv4/IPv6)

integer

Int32

long

Int64

float

Float32

double

Float64

When creating an index pattern, ensure the pattern matches the ClickHouse table name exactly and that a suitable time field (Date/DateTime64 or a field containing "time") is present.

Advanced features :

Sampling : For large result sets, CKibana can sample data to reduce ClickHouse resource consumption. Sample threshold is set to 5 million rows. The sampling factor is calculated as: Math.max(0.01, Double.parseDouble(String.format("%.5f", sampleParam.getSampleCountMaxThreshold() * 1.00 / sampleParam.getSampleTotalCount())))

Time round + cache : Round query time windows (e.g., 20 s) to increase cache hit rates. Configure with: curl --location --request POST 'localhost:8080/config/updateRoundAbleMinPeriod?roundAbleMinPeriod=20000' curl --location --request POST 'localhost:8080/config/updateUseCache?useCache=true'

Query monitoring & blacklist : All queries and their latency are recorded; a blacklist can block expensive queries. Enable monitoring with: curl --location --request POST 'localhost:8080/config/updateEnableMonitoring?enableMonitoring=true'

Query time limit : Restrict the maximum query time range (e.g., 10 days) to prevent heavy scans: curl --location --request POST 'localhost:8080/config/updateMaxTimeRange?maxTimeRange=864000000'

Keyword query : Preserve Elasticsearch's field.keyword exact‑match semantics by translating to ClickHouse equality (e.g., host="www.baidu.com" ).

Discover performance optimization : For large time ranges, CKibana first aggregates log counts per minute, then automatically narrows the detailed query window, dramatically reducing ClickHouse CPU usage.

Example use case: Nginx access logs – the ClickHouse table must place host as the first column in the ORDER BY clause for optimal query performance.

All configuration options are summarized in the following screenshot:

Since adopting CKibana, the company has migrated all Nginx and custom business logs from Elasticsearch to ClickHouse, cutting storage costs to less than 30% of the original, while maintaining Kibana’s familiar UI and achieving higher stability and fault tolerance.

The CKibana project is open‑source at https://github.com/TongchengOpenSource/ckibana . Contributions, stars, and issues are welcomed.