Common ELK Deployment Architectures and Practical Solutions for Log Management

ELK has become the most popular centralized logging solution, consisting of Beats, Logstash, Elasticsearch, and Kibana to provide real‑time log collection, storage, and visualization.

Core Components

Filebeat : a lightweight data shipper that can replace Logstash on application servers and output to Kafka, Redis, etc.

Logstash : a heavier data collection engine with many plugins for filtering, analyzing, and formatting logs.

Elasticsearch : a distributed search engine built on Apache Lucene, offering centralized storage, analysis, and powerful aggregation.

Kibana : a web‑based visualization platform for exploring data stored in Elasticsearch.

Common ELK Deployment Architectures

1. Logstash as Log Collector

Each application server runs a Logstash instance that collects, filters, and formats logs before sending them to Elasticsearch; Kibana visualizes the data. This approach consumes significant resources on the application servers.

2. Filebeat as Log Collector

Filebeat replaces Logstash on the application side, consuming far fewer resources; it is often paired with Logstash for further processing. This is currently the most common architecture.

3. Introducing a Caching Queue

Based on the Filebeat architecture, a Kafka (or other) message queue is added between Filebeat and Logstash to handle large data volumes, improve data safety, and balance load between Logstash and Elasticsearch.

4. Summary of the Three Architectures

The Logstash‑only architecture is rarely used due to its resource consumption. The Filebeat‑based architecture is the most popular, while the Kafka‑enhanced design is only necessary for very high‑volume scenarios.

If Logstash becomes busy, it signals Filebeat to slow down; once congestion clears, Filebeat resumes its normal rate.

Problems and Solutions

1. Multiline Log Merging

Logs that span multiple lines need to be merged into a single event. Use the multiline plugin in Filebeat or Logstash, depending on the deployment.

Filebeat multiline configuration:

filebeat.prospectors:
  -
    paths:
      - /home/project/elk/logs/test.log
    input_type: log
    multiline:
      pattern: '^\['
      negate: true
      match: after
output:
  logstash:
    hosts: ["localhost:5044"]

Key parameters:

pattern – regular expression to identify the start of a new log entry.

negate – true merges lines that do NOT match the pattern.

match – after merges to the previous line’s end.

Logstash multiline configuration:

input {
  beats {
    port => 5044
  }
}
filter {
  multiline {
    pattern => "%{LOGLEVEL}\s*]"
    negate => true
    what => "previous"
  }
}
output {
  elasticsearch {
    hosts => "localhost:9200"
  }
}

2. Replacing Kibana’s Default Timestamp with Log‑Embedded Time

Use the grok filter to extract the timestamp from the log message and the date filter to rewrite @timestamp.

filter {
  grok {
    match => ["message", "(?
%{YEAR}%{MONTHNUM}%{MONTHDAY}\s+%{TIME})"]
  }
  date {
    match => ["customer_time", "yyyyMMdd HH:mm:ss,SSS"]
    target => "@timestamp"
  }
}

3. Viewing Logs by Specific System Module in Kibana

Add a custom field (e.g., log_from ) in Filebeat to tag logs from different modules, then filter in Kibana.

filebeat.prospectors:
  -
    paths:
      - /home/project/elk/logs/account.log
    input_type: log
    multiline:
      pattern: '^\['
      negate: true
      match: after
    fields:
      log_from: account
  -
    paths:
      - /home/project/elk/logs/customer.log
    input_type: log
    multiline:
      pattern: '^\['
      negate: true
      match: after
    fields:
      log_from: customer
output:
  logstash:
    hosts: ["localhost:5044"]

Alternatively, use document_type and route to different Elasticsearch indices:

output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "%{type}"
  }
}

Conclusion

The article presented three ELK deployment architectures, highlighted the most widely used Filebeat‑based setup, and offered practical solutions for common logging challenges such as multiline merging, timestamp correction, and module‑level filtering.

Beyond centralized log search, ELK can also serve for application monitoring and server resource tracking.