Why Logging Matters: Building Distributed Log Operations & Observability

What Is a Log

A log is a time‑ordered record of events that captures what happened at a specific moment, providing precise system documentation. According to the APM definition, logs describe discrete (non‑continuous) events.

Logs are classified by severity levels such as FATAL, WARNING, NOTICE, DEBUG, and TRACE. Projects typically define a log level threshold; messages above that threshold are persisted.

When to Record Logs

In large‑scale web architectures, logs are a critical component. They record all system behaviors, aid in troubleshooting, performance optimization, and support product operations by providing user behavior data for business decisions. In security, logs reveal attacks like login failures or abnormal accesses, and they are essential for audit trails.

Value of Logs

Logs enable error localization, performance tuning, security analysis, and business insight. Their importance is evident across debugging, fault detection, intrusion detection, and audit tracking.

Distributed Log Operations

Modern micro‑service environments introduce challenges:

Different teams use varied languages and inconsistent log formats.

Rapid iteration leads to missing logs, incorrect levels, and difficulty extracting useful information.

Thousands of container instances across data centers make request tracing hard.

Manual log access via SSH and grep/awk becomes inefficient at scale due to large volumes, slow searches, and lack of multidimensional queries. Centralized log collection systems are needed to aggregate, manage, and access logs efficiently.

Why Tools Are Needed

Without tooling, engineers must log into each instance, search raw files, and manually extract data—a process that does not scale. Centralized solutions address storage, indexing, and fast querying.

Capabilities Required of Log‑Operation Platforms

Predict risks and bottlenecks before failures.

Notify promptly and locate issues during incidents.

Provide historical data for post‑mortem analysis.

Such platforms should support real‑time log collection, analysis, and storage, enabling rapid diagnosis, system operation, traffic stability monitoring, and business data analysis. Link tracing systems further enhance observability by recording request call chains.

APM and Observability

APM (Application Performance Management) is a methodology for observing, analyzing, and optimizing distributed systems. Monitoring (including alerts) forms part of the SLA framework, acting as a guard to detect and troubleshoot issues.

The APM ecosystem processes three data types—logs, metrics, and traces—across four stages: collection, processing, storage, and visualization, while addressing challenges like heterogeneous programs, diverse components, complete traceability, and timely sampling.

Observability, a key APM characteristic, rests on three pillars: Logging, Metrics, and Tracing.

Metrics and Prometheus

Metrics are aggregatable atomic data points (e.g., CPU usage, memory, response time, QPS). They are stored as time‑series in TSDBs and visualized via dashboards for health monitoring, capacity planning, and performance optimization.

Prometheus, a CNCF‑graduated open‑source monitoring solution, scrapes target metrics, stores them as time‑series, triggers alerts, and visualizes data through Grafana.

Logging and ELK

ELK (Elasticsearch, Logstash, Kibana) provides distributed search and log analysis. Elasticsearch offers a scalable, RESTful search engine; Logstash collects, transforms, and forwards logs; Kibana visualizes indexed data. ELK enables keyword search, multi‑dimensional log tracing, and visual dashboards.

Common optimizations include hot‑cold data separation, using Filebeat instead of Logstash for lightweight ingestion, and adding message queues for buffering and load smoothing.

Tracing, OpenTracing, and Apache SkyWalking

Tracing records request‑level call chains, allowing developers to pinpoint slow or faulty services. OpenTracing defines a vendor‑neutral API for integrating tracing libraries (e.g., Zipkin, Jaeger, SkyWalking). SkyWalking, an Apache top‑level project, supports Java, .NET, Node.js, and stores data in MySQL or Elasticsearch.

Combining Metrics, Logging, and Tracing

The three data types complement each other:

Metrics + logs enable event aggregation (e.g., traffic trends, error counts).

Tracing + logs provide detailed request information (inputs, outputs, intermediate logs).

Metrics + tracing reveal call frequencies and latency per service.

Integrating all three yields powerful fault‑diagnosis workflows: alerts → metric drill‑down → log inspection → trace analysis → resolution.

Batch Log Retrieval Tool (Go)

<code><span>package main</span>

<span>import (</span>
    <span>"fmt"</span>
    <span>"log"</span>
    <span>"os/exec"</span>
    <span>"runtime"</span>
    <span>"sync"</span>
<span>)</span>

<span>// Concurrent execution</span>
<span>var wg sync.WaitGroup</span>

<span>func main() {</span>
    runtime.GOMAXPROCS(runtime.NumCPU())
    instancesHost := getInstances()
    wg.Add(len(instancesHost))
    for _, host := range instancesHost {
        go sshCmd(host)
    }
    wg.Wait()
    fmt.Println("over!")
}

<span>// Execute query command</span>
<span>func sshCmd(host string) {</span>
    defer wg.Done()
    logPath := "/xx/xx/xx/"
    logShell := "grep 'FATAL' xx.log.20230207"
    cmd := exec.Command("ssh", "PasswordAuthentication=no", "ConnectTimeout=1", host, "-l", "root", "cd", logPath, "&&", logShell)
    out, err := cmd.CombinedOutput()
    fmt.Printf("exec: %s\n", cmd)
    if err != nil {
        fmt.Printf("combined out:\n%s\n", string(out))
        log.Fatalf("cmd.Run() failed with %s\n", err)
    }
    fmt.Printf("combined out:\n%s\n", string(out))
}

<span>// Retrieve instance IP list</span>
<span>func getInstances() []string {</span>
    return []string{"x.x.x.x", "x.x.x.x", "x.x.x.x"}
}
</code>

Deploy this script on a control machine with SSH key authentication to perform concurrent log queries across multiple hosts, with extensible parameters for target clusters, commands, concurrency limits, and output formatting.

Log Bad Smells

Unclear information reduces efficiency.

Non‑standard formats hinder readability and collection.

Insufficient logs lack key details for troubleshooting.

Redundant or meaningless logs waste resources.

Inconsistent log levels cause false alerts.

String concatenation instead of placeholders lowers maintainability.

Logging inside tight loops risks crashes.

Sensitive data not masked poses privacy risks.

Logs not rotated hourly complicate disk management.

Missing trace propagation prevents end‑to‑end tracing.

Log Good Cases

Quick problem localization.

Effective information extraction to understand root causes.

Clear view of system runtime state.

Aggregated key information reveals bottlenecks.

Logs evolve alongside project iterations.

Logging and collection do not impact normal system operation.

Conclusion

In the era of cloud‑native services, building an appropriate log‑operation platform that provides search, analysis, and alerting capabilities brings dormant server logs to life, facilitating data analysis, issue diagnosis, and system improvement. The practices described aim to help readers implement effective logging strategies in their own projects.