Master Elasticsearch ILM: Automate Index Lifecycle for Seamless Data Management

Maintaining legacy Elasticsearch clusters often leads to painful experiences such as midnight disk‑full alerts and manual handling of hundreds of indices.

ILM Overview and Benefits

ILM Overview : Index Lifecycle Management (ILM) policies automatically manage your indices according to performance, resilience, and retention requirements.

Benefits of ILM :

When an index reaches a certain size or document count, a new index is created.

New indices are created daily, weekly, or monthly while previous ones are archived.

Outdated indices are deleted to enforce data retention policies.

ILM Core Mechanism Dissection

Five ILM Phases Overview

Hot : high‑frequency read/write, stored on SSD/NVMe, ideal for real‑time log ingestion.

Warm : read‑only queries, stored on SAS, suitable for recent 7‑day log queries.

Cold : low‑frequency queries, stored on HDD, used for historical data archiving.

Frozen : ultra‑fast queries, stored on object storage, for compliance data retention.

Delete : index becomes inaccessible and is removed, used for expired data cleanup.

Actions Available in Each Phase

Hot: set priority, unfollow, transition, read‑only, shrink, force‑merge, searchable snapshot.

Warm: set priority, unfollow, read‑only, allocate, migrate, shrink, force‑merge.

Cold: set priority, unfollow, read‑only, searchable snapshot, allocate, migrate, freeze.

Frozen: searchable snapshot.

Delete: wait for snapshot, delete.

Reference: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/ilm-actions.html

Phase Transition Triggers

ILM moves an index to the next phase based on its "age".

All actions in the current phase must complete and the index must have existed longer than the minimum age for the next phase.

The default minimum age is zero, causing immediate transition after current actions finish.

Practical Configuration Guide

Kibana Panel Configuration

Management → Stack Management

Index Lifecycle Management → Create Policies

Hot phase → Advanced settings

Warm phase → Advanced settings

Delete phase

API Configuration

<code>curl -X PUT -u elastic -H 'Content-Type: application/json' -k https://10.97.82.201:9200/_ilm/policy/jiaxzeng -d '{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "10gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "readonly": {},
          "allocate": {
            "number_of_replicas": 0
          }
        }
      },
      "delete": {
        "min_age": "3d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}'
</code>

min_age indicates how long an index must stay in the previous phase; the example retains logs for a total of four days.

Conclusion

When indices manage their own lifecycle, operations engineers can finally get a good night’s sleep.