Full‑Link Monitoring: Concepts, Requirements, Architecture and Comparative Evaluation of APM Solutions

With the rise of micro‑service architectures, a single request often traverses many services deployed across thousands of servers and multiple data centers, making it essential to have tools that can observe system behavior and diagnose performance problems quickly.

Full‑link monitoring addresses this need by collecting trace data across service boundaries, similar to Google’s Dapper system. It enables rapid fault localization, dependency analysis, and capacity planning.

Objectives

Minimize probe performance overhead.

Maintain low code intrusion; the tracing system should be transparent to developers.

Provide good scalability for distributed deployment.

Deliver fast, multi‑dimensional data analysis.

Functional Modules

Instrumentation and Log Generation : client‑side, server‑side or bi‑directional instrumentation that records TraceId, SpanId, timestamps, protocol, IP/port, service name, latency, result, error info, and extensible fields.

Log Collection and Storage : agents on each host forward logs to a daemon, multi‑level collectors (pub/sub style) balance load, and aggregated data is stored for real‑time and offline analysis.

Analysis and Statistics : spans with the same TraceId are assembled into a timeline; parent‑child relationships reconstruct the call stack. Dependency metrics (strong, high, frequent) and both real‑time and batch analyses are supported.

Visualization and Decision Support : dashboards show stage‑wise latency, dependency graphs, and alerts.

Google Dapper Model

Span is the basic unit of work, identified by a 64‑bit ID and containing fields such as TraceID, Name, ID, ParentID, Annotations, and a Debug flag.

type Span struct {
    TraceID    int64 // identifies a complete request
    Name       string
    ID         int64 // current span ID
    ParentID   int64 // parent span ID, null for root
    Annotation []Annotation // timestamps and tags
    Debug      bool
}

Trace is a tree of spans representing a full request lifecycle from client request to server response.

Annotation records specific events (e.g., cs, sr, ss, cr) with timestamps, values, host information and duration.

type Annotation struct {
    Timestamp int64
    Value     string
    Host      Endpoint
    Duration  int32
}

Call Example

A user request hits front‑end service A, which calls services B and C; B returns immediately, while C interacts with D and E before responding, illustrating a complete trace with multiple spans.

Deployment Architecture

Agents generate trace logs, Logstash collects them into Kafka, Kafka feeds downstream consumers, Storm processes metrics into Elasticsearch, and HBase stores raw trace data for fast lookup by TraceID.

Solution Comparison

The three open‑source APM components examined are:

Zipkin (Twitter): collects and visualizes trace data via HTTP or MQ.

Pinpoint (Naver): Java‑centric APM with agents, collectors, and a web UI.

SkyWalking: Chinese APM supporting many middleware and frameworks.

Key comparison dimensions include probe performance, collector scalability, depth of call‑chain analysis, developer transparency, and topology visualization.

Probe Performance

Benchmarks with a Spring‑Boot application (500, 750, 1000 concurrent users) show SkyWalking has the smallest impact on throughput, Zipkin is moderate, and Pinpoint reduces throughput noticeably at higher loads.

Collector Scalability

Zipkin can scale horizontally by adding multiple server instances consuming from MQ; SkyWalking uses gRPC with single‑node or cluster modes; Pinpoint employs Thrift over UDP with cluster support.

Call‑Chain Analysis

SkyWalking provides the richest middleware coverage; Zipkin shows service‑level spans; Pinpoint records the most detailed data, including SQL statements.

Developer Transparency

Zipkin requires code changes or library integration; SkyWalking and Pinpoint use byte‑code instrumentation, making them non‑intrusive.

Topology Visualization

All three generate full call‑graph topologies; Pinpoint’s UI displays richer details (e.g., DB names), while Zipkin’s view is limited to service‑to‑service links.

Pinpoint vs. Zipkin

Pinpoint offers a complete APM stack with Java agents, HBase storage, and extensive UI features, but its ecosystem is smaller and integration for non‑Java languages is limited. Zipkin focuses on collector and storage, provides a flexible query API, and enjoys a larger community.

Tracing vs. Monitoring

Monitoring (system and application metrics) aims at anomaly detection and alerting, while tracing focuses on end‑to‑end request flow analysis to proactively identify performance bottlenecks.

References: Dapper translation, Pinpoint GitHub issues.