Extracting Nginx Logs with Logstash Using Grok Patterns

The article focuses on extracting Nginx logs through Logstash regular expressions, providing a complete workflow from Nginx log format definition to Kibana visualization.

1. Nginx log format configuration

The custom log format is defined as:

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

Example log entries are shown, illustrating typical access and error records.

2. Built‑in Grok extraction syntax

A Grok pattern that parses the log fields is provided:

%{IP:remote_addr} - (%{WORD:remote_user}|-) \[%{HTTPDATE:time_local}\] "%{WORD:method} %{NOTSPACE:request} HTTP/%{NUMBER}" %{NUMBER:status} %{NUMBER:body_bytes_sent} %{QS} %{QS:http_user_agent}

3. Logstash configuration and extraction steps

Several steps are described:

Using Kibana’s Grok debugger to test patterns.

Applying formal Grok syntax in Logstash pipelines.

Segmenting Nginx logs into fields.

Configuring Logstash to read the access log file and output to Elasticsearch.

The Logstash configuration file ( /etc/logstash/conf.d/logstash.conf ) looks like:

input {
  file {
    path => "/var/log/nginx/access.log"
  }
}
output {
  elasticsearch {
    hosts => ["http://192.168.20.41:9200", "http://192.168.20.42:9200"]
    user => "elastic"
    password => "hahashen"
    index => "sjgnginx-%{+YYYY.MM.dd}"
  }
}

After restarting Logstash, Kibana is used to create an index pattern for sjgnginx-* , enabling the parsed fields to be visualized.

Finally, the article shows screenshots of Kibana dashboards displaying the extracted Nginx log data and the final regular‑expression split results.