How to Securely Backup Elasticsearch with Snapshots and S3 Repositories

Background

Any software that stores data needs regular backups. Elasticsearch replicas provide high‑availability during node failures but cannot protect against catastrophic failures such as total data‑center loss. Therefore a full backup mechanism is required.

Snapshot and Restore

Elasticsearch offers a snapshot and restore API that lets you back up indices and cluster state. Snapshots serve as data copies that can be restored when problems occur.

Repository

To back up, you must create a repository. Elasticsearch supports several repository types:

Shared file system (type: "fs")

Read‑only URL (type: "url")

S3 (type: "s3")

HDFS (type: "hdfs")

Azure (type: "azure")

Google Cloud Storage (type: "gcs")

Note: S3, HDFS, Azure, and GCS require the corresponding plugins.

Single snapshot

A single snapshot represents a backup set stored in a repository and contains all selected indices.

Indices

A snapshot can contain multiple indices, which can be selected by pattern or individually.

S3 Plugin

To back up to S3 you need to install the repository‑s3 plugin on each node.

ES Cluster Plugin Installation and Configuration

1.1 Install repository‑s3 plugin

<code>./bin/elasticsearch-plugin install repository-s3</code>

1.2 Modify jvm.options

Add the following line to the end of the JVM options on every node:

<code>-Des.allow_insecure_settings=true</code>

1.3 Verify plugin

Restart Elasticsearch on each node and browse to

http://<em>es_ip</em>/_cat/plugins

to see installed plugins.

Creating a Huawei Cloud OBS Repository

Although the official documentation focuses on AWS S3, you can use Huawei Cloud OBS (or other object storage services) by configuring the repository type as "s3".

3.1 Create bucket and obtain access keys

3.2 Set OBS as snapshot repository

Use Kibana Dev Tools to create a repository named

es_s3_repository

with your bucket, access key, secret key, and endpoint. Enable compression.

<code>PUT _snapshot/es_s3_repository
{
  "type": "s3",
  "settings": {
    "access_key": "xxxxxxxxxxxxxxx",
    "secret_key": "xxxxxxxxxxxxxxxxx",
    "bucket": "es-centre",
    "endpoint": "xxxxxxxxxxxx",
    "compress": "true"
  }
}</code>

After execution, a true response indicates success.

3.3 Test snapshot creation (snapshot name: index_bak)

Create a snapshot for indices

index1,index2

:

<code>PUT /_snapshot/es_s3_repository/index_bak?wait_for_completion=true
{
  "indices": "index1,index2",
  "ignore_unavailable": true,
  "include_global_state": false,
  "metadata": {
    "taken_by": "kimchy",
    "taken_because": "backup before upgrading"
  }
}</code>

Common API commands

<code># List all repositories
GET /_snapshot/_all
# View snapshots in a repository
GET /_snapshot/es_s3_repository/_all?pretty
# List running snapshots
GET /_snapshot/_status?pretty
# Check status of a specific snapshot
GET /_snapshot/es_s3_repository/snapshot_2/_status?pretty
# Delete a snapshot
DELETE /_snapshot/es_s3_repository/snapshot_2</code>

3.4 Add snapshot policy in Kibana

Define a policy that creates a daily snapshot named with the date, scheduled at 00:10 (UTC+8).

3.5 Verify data on OBS