Observability Concepts and OpenTelemetry Architecture Overview

Observability treats a running software system as a black box unless it is actively observed. By collecting logs, metrics, and traces, engineers can move from passive fault handling to proactive fault detection, especially in micro‑service environments.

The three fundamental pillars are:

Logs – discrete log entries.

Metrics – aggregated numeric indicators.

Trace – request‑level call‑chain tracking.

A typical troubleshooting workflow starts with metrics alerts (e.g., CPU or memory usage > 80% or an instance down). An example Prometheus alert rule is shown below:

groups:
- name: AllInstances
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 1m
    annotations:
      title: 'Instance {{ $labels.instance }} down'
      description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minute.'
    labels:
      severity: 'critical'

After detecting an anomaly, trace information is used to locate the faulty node, and the associated trace_id is used to retrieve the relevant logs.

OpenTelemetry (OTel) unifies observability standards. Its history includes key milestones such as Google’s Dapper paper (2010), the rise of Kubernetes, Jaeger, the merger of OpenTracing and OpenCensus (2019), and the first GA release in 2021.

OTel’s architecture consists of three layers:

Client agents that instrument applications.

The Collector service that receives, processes, and exports data.

Back‑ends that store metrics, logs, and traces (e.g., VictoriaMetrics, StackRocks, Elasticsearch).

The OpenTelemetry Collector is the central component, composed of Receivers, Processors, and Exporters, supporting a wide variety of protocols and destinations.

For Java applications, the opentelemetry-java-instrumentation library provides a javaagent that automatically instruments most frameworks. Example usage:

# Java example
java -javaagent:path/to/opentelemetry-javaagent.jar \
    -jar myapp.jar

Tracing concepts include Spans (with fields such as SpanName, ParentID, SpanID, TraceID, start/end timestamps) and Span Kind (Client, Server, Internal, Producer, Consumer). Context propagation is essential for passing the trace_id across process boundaries via HTTP headers, gRPC metadata, or message properties, using Inject and Extract APIs.

Metrics in OTel follow standardized naming conventions and support Exemplars, which link a metric sample back to a trace, enabling fast navigation from a metric anomaly to the corresponding trace and logs.

Beyond OTel, the article introduces eBPF – a kernel‑level virtual machine that provides non‑intrusive, high‑performance, and safe instrumentation for Linux systems. While powerful, eBPF is more suited for platform‑level monitoring rather than fine‑grained business metrics.

Commercial platforms such as SigNoz and OpenObserve bundle the OTel collector and storage, offering unified UI and storage for logs, traces, and metrics, simplifying operations for small‑to‑medium teams.

In summary, organizations can either build a custom OTel stack or adopt integrated solutions like SigNoz or OpenObserve, depending on their operational requirements and resource constraints.