How to Collect Easysearch Logs with Filebeat OSS: A Step‑by‑Step Guide

Background and version selection

Easysearch is a Chinese search engine built on Elasticsearch 7.x and fully compatible with the Elasticsearch API, allowing direct use of Filebeat OSS 7.10.2 without extra plugins. Filebeat 7.10.2 is the last Apache‑2.0 licensed release and matches Easysearch 7.10.2 kernel, ensuring optimal compatibility.

Preparation before deployment

File upload considerations

Uploading the Filebeat tarball directly to /opt/ via SFTP fails with “Access is denied” because the directory belongs to root. The correct workflow is to upload to a user’s home directory, then move with sudo:

# upload to home
sftp> put filebeat-oss-7.10.2-linux-x86_64.tar.gz /home/your_user/
# move with sudo
sudo mv ~/filebeat-oss-7.10.2-linux-x86_64.tar.gz /opt/

Extract and verify directory structure

cd /opt
sudo tar -zxvf filebeat-oss-7.10.2-linux-x86_64.tar.gz
ls -al /opt/

After extraction the directory filebeat-7.10.2-linux-x86_64/ appears (the “‑oss” suffix is removed).

Configure filebeat.yml

Edit /opt/filebeat-7.10.2-linux-x86_64/filebeat.yml with the following key sections (full file shown in the original image): paths: /opt/easysearch/logs/*.log – wildcard collects all log files, including slow‑log and GC log. multiline.pattern: '^\[' – Easysearch logs start with “[”, so Java stack traces are merged. ssl.verification_mode: none – Easysearch uses a self‑signed certificate; TLS verification must be disabled. setup.ilm.enabled: false – Disable ILM to avoid errors when Filebeat tries to create an ILM policy. index: "easysearch-logs-%{+yyyy.MM.dd}" – Daily index naming for easier management.

Pre‑create index template

Before starting Filebeat, create an index template in Easysearch to define field types and prevent automatic mapping errors (e.g., @timestamp being mapped as text).

PUT /_index_template/easysearch-logs
{
  "index_patterns": ["easysearch-logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1
    },
    "mappings": {
      "properties": {
        "@timestamp": {"type": "date"},
        "message": {"type": "text"},
        "log.level": {"type": "keyword"},
        "log_type": {"type": "keyword"}
      }
    }
  }
}

The response {"acknowledged": true} indicates success. A warning about template name overlap is normal; the new template takes precedence.

Start Filebeat and verify

Run Filebeat in the foreground for debugging:

cd /opt/filebeat-7.10.2-linux-x86_64
./filebeat -e -c filebeat.yml

Expected log lines include version info, TLS disabled, configured paths, Harvester start messages for each log file, successful connection to Easysearch, and “Connection … established”. All *.log files should be harvested and marked as established.

Query a recent index to confirm data ingestion: GET easysearch-logs-2026.05.31/_search The returned document shows log.file.path and the full message, confirming correct ingestion.

Production recommendations

Run as a systemd service

# install as systemd service
sudo ./filebeat --path.config /opt/filebeat-7.10.2-linux-x86_64 service install
sudo systemctl enable filebeat
sudo systemctl start filebeat
sudo systemctl status filebeat

Separate inputs for different log types

Define multiple inputs in filebeat.yml with distinct log_type fields (main, slowlog, deprecation) to enable downstream filtering.

Index lifecycle management (ILM)

For long‑running collections, create a manual ILM policy that rolls over daily and deletes indices older than 30 days:

PUT /_ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {"max_age": "1d", "max_size": "5gb"}
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {"delete": {}}
      }
    }
  }
}

Common issues quick reference

SFTP upload “Access is denied” – /opt/ lacks write permission. Upload to home directory then sudo mv.

Startup “certificate signed by unknown authority” – Disable verification with ssl.verification_mode: none.

Index not created / data not stored – Pre‑create index template and set setup.ilm.enabled: false.

Multiline logs split – Add multiline.pattern: '^\[' to merge stack traces.

Harvester permission denied – Grant read permission: sudo chmod o+r /opt/easysearch/logs/*.log.

Conclusion

The article demonstrates the full pipeline: version selection → file deployment → configuration → template creation → start and verify. Key take‑aways are version alignment, permission handling, pre‑building templates, disabling ILM, and configuring multiline merging for Java stack traces. The solution works out‑of‑the‑box without commercial licenses and is suitable for long‑term production use.