Centralized Log Collection and Monitoring with Filebeat and Graylog

When a company runs many services across test and production environments, collecting logs becomes essential; the article compares using Nginx for log exposure versus a dedicated log collection system like ELK, and recommends Graylog as a simpler alternative.

Filebeat Overview – Filebeat monitors specified log files or directories, spawns harvesters for each file, forwards new log entries to a configured output (e.g., Graylog), and supports lightweight deployment compared to Logstash.

Filebeat Configuration Example

# Configure input sources
filebeat.config.inputs:
  enabled: true
  path: ${path.config}/inputs.d/*.yml
# Enable modules
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
# Output to Graylog (via Logstash input)
output.logstash:
  hosts: ["11.22.33.44:5500"]
processors:
  - add_host_metadata: ~
  - rename:
      fields:
        - from: "log"
          to: "message"
  - add_fields:
      target: ""
      fields:
        token: "0uxxxxaM-1111-2222-3333-VQZJxxxxxwgX "

An additional inputs.d YAML file can define per‑service log paths, tags, multiline patterns, and filters, for example:

# Collect log type
- type: log
  enabled: true
  paths:
    - /var/log/supervisor/app_escape_worker-stderr.log
    - /var/log/supervisor/app_escape_prod-stderr.log
  symlinks: true
  include_lines: ["WARNING", "ERROR"]
  tags: ["app","escape","test"]
  multiline.pattern: '^\[?[0-9]...{3}'
  multiline.negate: true
  multiline.match: after

Graylog Overview – Graylog is an open‑source log aggregation and analysis platform that stores logs in Elasticsearch, configuration data in MongoDB, and provides a web UI for searching, streaming, and pipeline processing.

Key Graylog concepts include Inputs (log sources), Extractors (field parsing), Streams (routing), Index Sets (storage), and Pipelines (advanced processing). A sample pipeline rule to discard debug messages:

rule "discard debug messages"
when
  to_long($message.level) > 6
then
  drop_message();
end

Deployment – The article shows how to install Filebeat via DEB/RPM, Docker, or source, and how to deploy Graylog using Docker‑Compose with MongoDB and Elasticsearch services. Example Docker‑Compose snippet:

version: "3"
services:
  mongo:
    image: "mongo:3"
    container_name: graylog_mongo
    restart: on-failure
    volumes:
      - "./mongodb:/data/db"
    networks:
      - graylog_network
  elasticsearch:
    image: "elasticsearch:6.8.5"
    container_name: graylog_es
    restart: on-failure
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - ES_JAVA_OPTS=-Xms512m -Xmx5120m
    volumes:
      - "./es_data:/usr/share/elasticsearch/data"
    networks:
      - graylog_network
  graylog:
    image: "graylog/graylog:3.3"
    container_name: graylog_web
    restart: on-failure
    ports:
      - "9000:9000"   # web UI
      - "5044:5044"   # Filebeat input
      - "12201:12201" # GELF TCP
      - "12201:12201/udp" # GELF UDP
      - "1514:1514"   # Syslog TCP
      - "1514:1514/udp" # Syslog UDP
    environment:
      - GRAYLOG_PASSWORD_SECRET=zscMb65...FxR9ag
      - GRAYLOG_ROOT_PASSWORD_SHA2=77e29e0f...557515f
      - GRAYLOG_HTTP_EXTERNAL_URI=http://11.22.33.44:9000/
      - GRAYLOG_TIMEZONE=Asia/Shanghai
    volumes:
      - "./graylog_journal:/usr/share/graylog/data/journal"
    depends_on:
      - mongo
      - elasticsearch
    networks:
      - graylog_network
networks:
  graylog_network:
    driver: bridge

Spring Boot Integration – To forward application logs to Graylog, add the logback-gelf dependency and configure logback.xml with a GELF UDP appender pointing to the Graylog host and port, optionally setting fields such as app_name .

<appender name="GELF" class="de.siegmar.logbackgelf.GelfUdpAppender">
  <graylogHost>ip</graylogHost>
  <graylogPort>12201</graylogPort>
  <maxChunkSize>508</maxChunkSize>
  <useCompression>true</useCompression>
  <encoder class="de.siegmar.logbackgelf.GelfEncoder">
    <includeRawMessage>false</includeRawMessage>
    <includeMarker>true</includeMarker>
    <includeMdcData>true</includeMdcData>
    <includeLevelName>true</includeLevelName>
    <shortPatternLayout class="ch.qos.logback.classic.PatternLayout">
      <pattern>%m%nopex</pattern>
    </shortPatternLayout>
    <fullPatternLayout class="ch.qos.logback.classic.PatternLayout">
      <pattern>%d - [%thread] %-5level %logger{35} - %msg%n</pattern>
    </fullPatternLayout>
    <staticField>app_name:austin</staticField>
  </encoder>
</appender>

Finally, the article demonstrates how to search logs in Graylog using simple queries, field‑specific filters, and logical operators, enabling developers and operators to quickly locate issues across distributed services.