Integrating SkyWalking with ELK for Distributed Trace ID Logging

1. Background

When building a logging platform we adopted SkyWalking + ELK, but discovered that the ELK logs did not contain a Trace ID, making it impossible to trace the full error chain across services.

Trace ID is a unique identifier used in distributed tracing to link all logs and performance data belonging to a single request across multiple micro‑service nodes.

Why does SkyWalking provide trace IDs while ELK does not?

2. Relationship Between SkyWalking and ELK

SkyWalking: An Application Performance Monitoring (APM) system that offers distributed tracing, service performance analysis, and multi‑dimensional monitoring. It supports automatic code instrumentation to trace calls and metrics.

ELK: A centralized log management and analysis stack (Elasticsearch + Logstash + Filebeat) with Kibana for visual log search.

2.1 SkyWalking

SkyWalking includes built‑in tracing and provides a UI (SkyWalking‑UI) for viewing logs and trace data.

Overall architecture of SkyWalking:

Tracing component: Collects trace data from applications and sends it to the SkyWalking OAP server (supports SkyWalking, Zipkin, Jaeger, etc.).

SkyWalking OAP server: Receives trace data, analyses it, stores it in external storage, and provides query capabilities.

Storage: Persists trace data; supports ES, MySQL, ShardingSphere, TiDB, H2, etc.

SkyWalking UI: Web interface for visualising trace and metric data.

2.2 ELK Log Platform

The overall architecture of the ELK stack is shown below:

Data flow:

Beats (Filebeat): Deployed on the application side to collect logs and forward them to Logstash.

Logstash: Filters, transforms, and forwards log data to Elasticsearch.

Elasticsearch: Stores log data and builds indexes for fast search.

Kibana: Provides visual query and analysis of logs.

2.3 Similarities Between SkyWalking and ELK

Both can collect logs.

Both provide a UI for log query and visualisation.

3. Can We Use Only SkyWalking?

SkyWalking excels at service performance analysis and tracing, but has some drawbacks.

3.1 Limitations in Collection Methods

SkyWalking agents collect tracing data via SDK or Agent for Java, Go, Node, .NET, etc., using gRPC to send data to the OAP server. For Nginx or MySQL logs, custom scripts or Lua code are required.

3.2 Limitations in Data Visualisation

Trace visualisation is intuitive, but log exploration capabilities are weak compared with Kibana’s rich charts and filters.

Search and highlighting features are far less powerful than Kibana’s.

Below are screenshots of SkyWalking and Kibana visualisation interfaces:

4. Can ELK Achieve Traceability on Its Own?

ELK does not provide native trace IDs; additional tools or configurations are needed. Common approaches include:

Embedding Trace ID via SkyWalking Agent.

Adding Trace ID to MDC (Mapped Diagnostic Context) in the application.

Using Kibana’s “View surrounding documents” feature (limited accuracy).

4.1 Embedding Trace ID with SkyWalking

Use SkyWalking’s TraceIdPatternLogbackLayout to inject the Trace ID into log messages.

4.1.1 Configuration Example

In logback-spring.xml configure the console appender to use the custom layout and include [%tid] in the pattern:

<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <!-- Define a layout that prints TraceId -->
    <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
        <pattern>${CONSOLE_LOG_PATTERN:-%clr(%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) [%tid] %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}</pattern>
    </layout>

    <root level="debug">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

When the application runs, the console will print logs with the Trace ID, e.g.:

Filebeat and Logstash then forward these logs to Elasticsearch, where Kibana can query by Trace ID.

4.1.2 Principle

Context propagation: Trace ID is passed between services via HTTP headers or other mechanisms.

Log integration: SkyWalking uses byte‑code enhancement or automatic proxies to generate and manage Trace IDs at runtime.

Configuration flexibility: Developers can customise log formats (e.g., via logback.xml ) to include or exclude the Trace ID.

4.2 MDC Approach

Generate a random ID, put it into MDC, and reference it in the log pattern. Example:

MDC.put("traceId", UUID.randomUUID().toString());

MDC stores data per thread, allowing any code in the same thread to access the Trace ID.

For a deeper explanation see the linked article on micro‑service tracing.

From version 1.5, building a micro‑service framework – trace ID

4.3 Kibana’s “Surrounding Documents” Feature

Kibana can display logs that are temporally close to a selected log by clicking the View surrounding documents button.

However, this method may not reliably identify logs that belong to the same request context because many unrelated logs can appear in the same time window.

High volume of similar logs in a short period.

Interleaved logs from other requests make filtering difficult.

5. Conclusion

SkyWalking and ELK each play important roles in APM and log management. Although ELK does not natively support distributed tracing, integrating it with SkyWalking allows the two systems to complement each other and improve observability in micro‑service architectures.