Real-time Monitoring Dashboard for Logistics Supply Chain: Architecture, Data Processing, and Stability Practices

Overview

The logistics supply‑chain dashboard has evolved over two years to become a mature, stable system that handles complex data flows and stringent reliability requirements.

This document outlines the framework and experiences of building real‑time monitoring indicators for a complex logistics supply‑chain scenario, leveraging big‑data technologies to achieve high availability and low latency.

1. Data Processing Chain and OLAP Engine

Using the mature stream‑processing engine Flink together with ClickHouse, a foundational model is built, and dual streams with EasyData enable one‑click switching.

2. Metric Consistency

Processing and presentation are decoupled; atomic metrics at the warehouse level can be rolled up to regional and national dimensions, ensuring both dimensional and temporal consistency across data versions, aided by cache tables for various business scenarios.

3. Stability Design

Key stability aspects include data‑pipeline reliability, service fallback mechanisms, metric accuracy validation, and recomputation strategies.

4. Architecture

The system is organized into five layers: model processing, data processing, warehouse‑level metric processing, regional metric processing, and presentation, as illustrated in the architecture diagram.

5. Metric Layering and Consistency

All metrics share a single processing logic; after a primary task completes, warehouse metrics are aggregated to higher‑level metrics, maintaining consistency. Cache tables reduce data volume and improve query performance.

6. Stability Design Details

Long data pipelines introduce instability, especially in Flink and ClickHouse components. Historical issues such as partition overload, rename delays, network failures, message loss, checkpoint failures, and complex processing logic are documented.

To mitigate these, a dual‑stream physical isolation is employed for the Flink‑ClickHouse link, allowing instant failover to a backup stream.

7. Extensibility Design

Configuration‑driven adaptability via UCC enables flexible business requirements, reducing development effort and simplifying issue diagnosis and recovery. Configurable features include 28‑hour mode, threshold switches, auto‑refresh whitelists, historical date selection, and recomputation mechanisms.

Example parameter configuration:

{
  "thresholdEnable": "false",
  "upperLimit": "1.6d",
  "lowerLimit": "0.6d",
  "zyShowFlag": true,
  "swShowFlag": true,
  "jjShowFlag": true,
  "wdShowFlag": true,
  "todayTradeCleanRateShowFlag": true,
  "promotionTradeCleanRateShowFlag": true,
  "isDebug": true,
  "isCacheOn": true,
  "isWriteMinuteAndHour": true,
  "isMinuteWrite": true,
  "isHourWrite": true,
  "isMinuteNotice": false,
  "isHourNotice": false
}

Strategy configuration example:

{
  "sTime": "2023-06-17 00:00:00",
  "eTime": "2023-06-17 19:59:59",
  "tbSTime": "2022-06-17 00:00:00",
  "tbETime": "2022-06-17 19:59:59",
  "hbSTime": "2022-11-10 00:00:00",
  "hbETime": "2022-11-10 19:59:59",
  "showType": "24h",
  "special24hCompDateStr": "2022-06-17",
  "specialCompDateStr": ""
}

8. Data Monitoring

Multiple verification and monitoring methods ensure data accuracy, including automated front‑end model screenshots, minute‑level request logging, Grafana dashboards for real‑time monitoring, and test models built with JD Power to validate metric calculations.

The article concludes with an invitation for further discussion and hopes to inspire developers working on large‑screen dashboards.