Real‑Time Monitoring Dashboard for Logistics Supply Chain: Architecture, Data Modeling, and Stability Design

Overview

In logistics supply‑chain dashboards, the data chain is long, the number of metrics is high (170+), and stability and accuracy requirements are strict; after two years of iteration the system is mature and stable.

The article describes a framework and experience for building real‑time monitoring dashboards using big‑data technologies, focusing on high availability and low latency.

Solution

The solution relies on a Flink‑based data processing pipeline and an OLAP engine built on ClickHouse, with a dual‑stream mechanism and EasyData for one‑click switching.

2.1 Data Model Storage Selection

Comparing Elasticsearch and ClickHouse for the large‑scale metric storage:

Comparison Item

Elasticsearch

ClickHouse

Implementation Principle

Distributed search engine based on Lucene with sharding and replication.

MPP‑based distributed ROLAP columnar DBMS with vectorized execution.

Write Performance

Medium, with latency issues.

High, throughput >5× Elasticsearch.

Query Performance

Medium.

High, 5‑30× faster.

Multi‑Table Join

Not supported.

Supported.

Server Cost

High.

Disk usage 1/3‑1/30 of ES, lower memory and CPU.

SQL Query

Not supported.

Supported.

High Concurrency

Good, can reach ten‑thousands QPS after tuning.

Officially recommended QPS 100.

Full‑Text Search

Supported.

Not supported.

For scenarios not requiring high concurrency or full‑text search, ClickHouse is preferred; otherwise Elasticsearch or a ClickHouse+cache layer can be used.

2.2 Overall Architecture

The system is divided into five layers: model processing, data processing, single‑warehouse metric processing, regional metric processing, and presentation. This reduces code complexity, improves maintainability, and enhances metric reuse.

2.3 Metric Layer and Consistency Design

All metrics share a single processing logic; after a primary task finishes, single‑warehouse metrics are rolled up to regional and national levels, ensuring dimensional and temporal consistency.

2.4 Stability Design

Stability challenges stem from the long data chain, especially Flink and ClickHouse. Historical issues include partition overload, rename delays, network failures, checkpoint failures, data loss, and operator back‑pressure. To mitigate, a dual‑stream physical isolation is used, allowing one‑click failover.

2.5 Extensibility Design

Configuration‑driven adaptability (UCC) supports 28‑hour/4‑hour modes, threshold switches, auto‑refresh white‑list, historical date selection, and re‑calculation mechanisms, reducing development cost and simplifying troubleshooting.

{
  "thresholdEnable": "false",
  "upperLimit": "1.6d",
  "lowerLimit": "0.6d",
  "zyShowFlag": true,
  "swShowFlag": true,
  "jjShowFlag": true,
  "wdShowFlag": true,
  "todayTradeCleanRateShowFlag": true,
  "promotionTradeCleanRateShowFlag": true,
  "isDebug": true,
  "isCacheOn": true,
  "isWriteMinuteAndHour": true,
  "isMinuteWrite": true,
  "isHourWrite": true,
  "isMinuteNotice": false,
  "isHourNotice": false
}

{
    "sTime": "2023-06-17 00:00:00",
    "eTime": "2023-06-17 19:59:59",
    "tbSTime": "2022-06-17 00:00:00",
    "tbETime": "2022-06-17 19:59:59",
    "hbSTime": "2022-11-10 00:00:00",
    "hbETime": "2022-11-10 19:59:59",
    "showType": "24h",
    "special24hCompDateStr": "2022-06-17",
    "specialCompDateStr": ""
}

2.6 Data Monitoring

Multiple verification and monitoring methods ensure data accuracy: front‑end automated model screenshots, automated packet capture, minute‑level data persistence with Grafana dashboards, and JD Power test models for detailed metric validation.