Using Filebeat and Graylog for Centralized Log Collection and Monitoring

When many services and environments are deployed internally, log inspection becomes a critical requirement. The article introduces two main solutions: using Nginx to expose logs or adopting a dedicated log‑collection service such as ELK or Graylog. Graylog is chosen for its simplicity, Elasticsearch storage, MongoDB caching, throttling, and user‑friendly UI.

Filebeat Overview

Filebeat is a lightweight log shipper that monitors specified log directories or files, reads new entries, and forwards them to Elasticsearch, Logstash, or Graylog. Its workflow includes starting prospectors, spawning harvesters for each file, sending harvested data to a spooler, and finally delivering the aggregated events to the configured output.

Key points:

Filebeat is lighter than Logstash and suitable for environments with limited resources.

Configuration files reside in /etc/filebeat/filebeat.yml for rpm/deb installations, with additional inputs.d and modules.d directories for modular settings.

Filebeat Configuration Example

# 配置输入来源的日志信息
# 我们合理将其配置到了 inputs.d 目录下的所有 yml 文件
filebeat.config.inputs:
  enabled: true
  path: ${path.config}/inputs.d/*.yml
  # 若收取日志格式为 json 的 log 请开启此配置
  # json.keys_under_root: true

# 配置 Filebeat 需要加载的模块
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

# 配置将日志信息发送的地址上面
output.logstash:
  hosts: ["11.22.33.44:5500"]

processors:
  - add_host_metadata: ~
  - rename:
      fields:
        - from: "log"
          to: "message"
  - add_fields:
      target: ""
      fields:
        token: "0uxxxxaM-1111-2222-3333-VQZJxxxxxwgX "

Example of an inputs.d YAML file that defines a log type, file paths, tags, and multiline handling:

# 收集的数据类型
- type: log
  enabled: true
  # 日志文件的路径地址
  paths:
    - /var/log/supervisor/app_escape_worker-stderr.log
    - /var/log/supervisor/app_escape_prod-stderr.log
  symlinks: true
  # 包含的关键字信息
  include_lines: ["WARNING", "ERROR"]
  # 打上数据标签
  tags: ["app", "escape", "test"]
  # 防止程序堆栈信息被分行识别
  multiline.pattern: '^\[?[0-9]...{3}'
  multiline.negate: true
  multiline.match: after

# 需要配置多个日志时可加多个 type 字段
- type: log
  enabled: true
  ...

Graylog is an open‑source log aggregation, analysis, and alerting platform. Its core components are Elasticsearch (for persistent storage and search), MongoDB (for Graylog configuration), and the Graylog server itself (providing the web UI and input handling). The article shows both single‑node and clustered deployment diagrams.

Key Graylog concepts covered:

Inputs – sources of log data, configurable with extractors for field transformation.

Streams – logical groupings of logs that can route to separate Elasticsearch index sets.

Extractors – field parsers applied per input.

Pipelines – server‑side processing rules; an example rule discards messages with level > 6.

Sidecar – lightweight collector that pulls configuration from Graylog and supports Filebeat, NXLog, and Winlogbeat.

Pipeline rule example (discard debug messages):

rule "discard debug messages"
when
  to_long($message.level) > 6
then
  drop_message();
end

Deployment Steps

Filebeat can be installed via:

Deb package: # Ubuntu (deb) $ curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.8.1-amd64.deb $ sudo dpkg -i filebeat-7.8.1-amd64.deb $ sudo systemctl enable filebeat $ sudo service filebeat start

Docker container: docker run -d --name=filebeat --user=root \ --volume "./filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \ --volume "/var/lib/docker/containers:/var/lib/docker/containers:ro" \ --volume "/var/run/docker.sock:/var/run/docker.sock:ro" \ docker.elastic.co/beats/filebeat:7.8.1 filebeat -e -strict.perms=false \ -E output.elasticsearch.hosts=["elasticsearch:9200"]

Graylog is deployed using Docker‑Compose. After generating a password secret and root password hash, a docker-compose.yml file defines three services (MongoDB, Elasticsearch, Graylog) and exposes the necessary ports (9000 for UI, 5044 for Filebeat, 12201 for GELF, 1514 for Syslog). Example snippet:

version: "3"
services:
  mongo:
    restart: on-failure
    container_name: graylog_mongo
    image: "mongo:3"
    volumes:
      - "./mongodb:/data/db"
    networks:
      - graylog_network

  elasticsearch:
    restart: on-failure
    container_name: graylog_es
    image: "elasticsearch:6.8.5"
    volumes:
      - "./es_data:/usr/share/elasticsearch/data"
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx5120m"
    networks:
      - graylog_network

  graylog:
    restart: on-failure
    container_name: graylog_web
    image: "graylog/graylog:3.3"
    ports:
      - 9000:9000   # Web UI
      - 5044:5044   # Filebeat input
      - 12201:12201 # GELF TCP
      - 12201:12201/udp
      - 1514:1514   # Syslog TCP
      - 1514:1514/udp
    environment:
      - GRAYLOG_PASSWORD_SECRET=zscMb65...FxR9ag
      - GRAYLOG_ROOT_PASSWORD_SHA2=77e29e0f...557515f
      - GRAYLOG_HTTP_EXTERNAL_URI=http://11.22.33.44:9000/
      - GRAYLOG_TIMEZONE=Asia/Shanghai
    networks:
      - graylog_network
    depends_on:
      - mongo
      - elasticsearch

networks:
  graylog_network:
    driver: bridge

To forward Docker container logs to Graylog, the GELF log driver can be used:

# Docker run with GELF driver
docker run --rm=true \
  --log-driver=gelf \
  --log-opt gelf-address=udp://11.22.33.44:12201 \
  --log-opt tag=myapp \
  myapp:0.0.1

Or configure it in docker‑compose.yml for services such as Redis:

services:
  redis:
    restart: always
    image: redis
    logging:
      driver: gelf
      options:
        gelf-address: udp://11.22.33.44:12201
        tag: "redis"

The article concludes with screenshots of the Graylog UI, a reminder to join the technical learning group, and a call‑to‑action to like the post if it was helpful.