How Volcano Engine’s DataLeap Platform Transforms Data Service Management

Platform Overview

Volcano Engine's DataLeap Data Service Platform provides a one‑stop API service platform that enables users to quickly turn data into services, offering full lifecycle management of API creation, governance, operation and sharing.

It ensures high reliability and security while building a unified data service gateway for various business lines, facilitating data sharing and bridging data and applications.

It solves problems of data understanding difficulty, heterogeneity, duplication, and audit/operation challenges, achieving unified and diversified data services and maximizing data application value.

Domain Introduction

Users can create projects, data sources, physical/logic tables, ready times, APIs, API orchestrations, and App applications on the platform.

Data sources contain connection info; physical tables are underlying source tables; logic tables enhance physical tables with monitoring, master‑slave switching, authorization, etc. Ready time is the latest available date. APIs and API orchestrations are data export points, and App identifies consumer identity.

Architecture Overview

The platform consists of platform services and engine services. It focuses on quality, efficiency and cost, providing high‑quality data service channels, monitoring, alerts, disaster recovery, and more.

Key capabilities include monitoring and alerts, permission control (project‑level and API‑level), rate limiting at data source, API, and API‑PSM levels, data source switching, logical table master‑slave switching, API version management, intelligent Q&A, query analysis, and operational dashboards.

API Orchestration Introduction

API orchestration allows combining multiple APIs to meet complex business logic without writing extensive code. It reduces development and maintenance costs.

Examples: selecting real‑time or offline data based on readiness time, integrating heterogeneous data sources, and performing data processing such as personalized product recommendations.

Principles

Orchestration builds a DAG (Directed Acyclic Graph) of nodes and directed edges. Nodes are basic data processing units; edges represent data flow.

Only one start node and one end node exist. Nodes can dynamically schedule downstream nodes based on parameters and conditions. Metadata is stored per node to avoid large keys.

Scheduling Design

Each node becomes a task with in‑degree tracking. The scheduler validates DAG legality, performs node‑level timeout, rate limiting, retries, and logs.

Tasks are placed in queues for concurrent execution, and the scheduler is decoupled from tasks.

Node Types

Node Type

Meaning

Dependency

Start Node

Entry point, receives request parameters

Input parameters

End Node

Output node, returns API response

Upstream output

API Node

Selects script‑style or wizard‑style APIs in the project

Project APIs

Function Node

Supports Faas or custom functions (Python, JAR)

Faas, Python, JAR

Branch Node

Conditional routing based on upstream data

Upstream input & condition

Programming Node

Executes Python scripts for data processing

Tos, Minio, Lego

Merge Node

Supports append, merge, join of multiple upstream nodes

All upstream outputs

Best Practices – Scenario 1: Live‑Screen Minute Data Hot/Cold Separation

Problem: High‑frequency HGETALL queries on Abase cause scan‑queue overload and frequent failures.

Solution: Use API orchestration to route hot data to Redis and cold data to Abase based on a 7‑day freshness flag.

Result: HGETALL queries dropped by 77.6%, dramatically improving live‑screen stability.

Best Practices – Scenario 2: Real‑Time/Offline Switch for E‑commerce Daily Metrics

Problem: Distributed real‑time/offline switch logic in downstream services leads to high iteration cost and scattered code.

Solution: Use branch nodes to read ready‑time metadata and decide between real‑time and offline APIs; merge nodes combine the results, providing a visual, low‑code workflow.

Result: Clear visual processing, low iteration cost, and comprehensive lineage for faster troubleshooting.

Conclusion

DataLeap’s API orchestration provides a visual, low‑code way to build, manage, and evolve data services, improving reliability, reducing operational overhead, and enabling rapid adaptation to storage or business changes.