Building and Using an ELK Real‑Time Log Analysis Platform for Spring Boot and Nginx

Log analysis is essential for troubleshooting micro‑service architectures, but logs are scattered across many machines; a unified real‑time platform like ELK dramatically improves efficiency.

ELK Overview : ELK consists of three open‑source components—Elasticsearch for storage and search, Logstash for data collection and processing, and Kibana for visualization.

Logstash collects logs from various sources, applies filters, and forwards them to destinations. Its pipeline includes input, filter, and output stages.

Elasticsearch is a distributed RESTful search and analytics engine offering fast, scalable, and flexible querying.

Kibana provides a browser‑based UI to create dashboards and explore Elasticsearch data.

Implementation Plan : Deploy Logstash shipper on each service machine to push logs to a Redis queue, use a separate Logstash indexer to read from Redis, parse logs, and store them in Elasticsearch, then visualize with Kibana.

Platform Setup (Ubuntu VM): install JDK ≥ 1.7, download Logstash, Elasticsearch, and Kibana packages, and extract them.

Install Logstash :

tar -xzvf logstash-7.3.0.tar.gz

cd logstash-7.3.0

bin/logstash -e 'input { stdin {} } output { stdout {} }'

Successful start is confirmed by log output.

Install Elasticsearch :

tar -xzvf elasticsearch-7.3.0-linux-x86_64.tar.gz

cd elasticsearch-7.3.0

bin/elasticsearch

Common startup issues include insufficient JVM memory (adjust config/jvm.options ) and running as root (use a non‑root user). Verify with curl http://localhost:9200 .

Install Kibana :

tar -xzvf kibana-7.3.0-linux-x86_64.tar.gz

cd kibana-7.3.0-linux-x86_64

./bin/kibana

Access http:// ip :5601 to confirm the UI.

Integrate Spring Boot : add a logback.xml configuration that writes logs to /log/sb-log.log , package the application, and deploy it on Ubuntu.

Shipper Logstash (Logback) configuration (excerpt):

input { file { path => "/log/sb-log.log" } }
output { redis { host => "10.140.45.190" port => 6379 db => 8 data_type => "channel" key => "logstash_list_0" } }

Indexer Logstash (Logback) configuration parses the log with a Grok filter and writes to Elasticsearch:

input { redis { host => "192.168.142.131" port => 6379 db => 8 data_type => "channel" key => "sb-logback" } }
filter { grok { match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NOTSPACE:threadName}\] %{LOGLEVEL:level} %{DATA:logger} %{NOTSPACE:applicationName} -" } } }
output { elasticsearch { hosts => "localhost:9200" index => "logback" } stdout { } }

Integrate Nginx : configure a second Logstash shipper to read /var/log/nginx/access.log and a corresponding Grok pattern to extract fields such as IP, method, URL, status, etc.

Combined Indexer Configuration uses the type field to route logback and nginx inputs through separate filter and output blocks.

Background Execution : Use Supervisor to run Elasticsearch, Logstash, and Kibana as daemon services. Example supervisord.conf sections:

[program:elasticsearch]
environment=JAVA_HOME="/usr/java/jdk1.8.0_221/"
command=/home/elk/elk/elasticsearch/bin/elasticsearch

[program:logstash]
command=/home/elk/elk/logstash/bin/logstash -f /home/elk/elk/logstash/indexer-logstash.conf

[program:kibana]
command=/home/elk/elk/kibana/bin/kibana

Reload Supervisor with sudo supervisorctl reload to start all components automatically on boot.

Conclusion : The guide demonstrates building an ELK stack, ingesting Spring Boot Logback and Nginx access logs, and managing the services in the background, providing a practical log‑analysis solution for micro‑service environments.