Log Collection Solution: Filebeat + Graylog – Configuration and Deployment Guide

The article introduces the need for centralized log collection when managing multiple services and environments, comparing Nginx, ELK, and Graylog as possible solutions and concluding that Graylog combined with Filebeat offers a simple, lightweight, and effective approach.

1. Filebeat Tool Overview – Filebeat is a log shipper that monitors specified log files or directories, reads new entries, and forwards them to Elasticsearch, Logstash, or Graylog. It uses prospectors and harvesters to process each file and supports throttling, tagging, and multiline patterns.

2. Filebeat Configuration – The core configuration resides in /etc/filebeat/filebeat.yml (or platform‑specific paths). Key sections include filebeat.config.inputs to load all inputs.d/*.yml files, module definitions, template settings, and output configuration (e.g., output.logstash.hosts ). Example input configuration shows how to define log paths, include specific lines, add tags, and handle multiline logs.

3. Graylog Service Overview – Graylog is an open‑source log aggregation, analysis, and alerting platform. Its architecture consists of Elasticsearch for storage, MongoDB for configuration, and Graylog itself for the web UI and processing. Core concepts include Inputs (log sources), Extractors (field parsing), Streams (log routing), Index Sets, Pipelines, and Sidecar agents.

4. Graylog Components – The article lists essential components such as Dashboards, Searching, Alerts, Inputs, Extractors, Streams, Indices, Outputs, Pipelines, Sidecar, Lookup Tables, and Geolocation, describing their functions and main features.

5. Deployment Steps – Detailed instructions cover installing Filebeat via RPM/DEB, Docker, or Kubernetes, and deploying Graylog using Docker‑Compose. Sample Docker‑Compose files define MongoDB, Elasticsearch, and Graylog services with necessary environment variables (password secret, root password SHA2, timezone, ports). Additional Docker run commands illustrate how to send container logs to Graylog using the GELF driver.

6. Practical Code Snippets – The guide includes full YAML configurations for Filebeat inputs, module settings, and Graylog Docker‑Compose, as well as pipeline rules to discard debug messages and Docker commands to enable GELF logging with tags.

Overall, the article serves as a step‑by‑step tutorial for setting up a robust, scalable log collection pipeline with Filebeat and Graylog, suitable for both development and production environments.