Kubernetes Cluster Backup: Comparing etcd Snapshots, Resource‑Level Backups, and Velero Usage

When operating a Kubernetes cluster, accidental deletion of a namespace can cause loss of all resources, forcing a manual re‑deployment. To avoid this, administrators use backup tools that can capture the entire cluster state or individual resources.

etcd backup creates a global snapshot of the cluster state, which can be restored to a specific point in time but cannot target individual objects. A typical backup script looks like:

#!/usr/bin/env bash
date
CACERT="/opt/kubernetes/ssl/ca.pem"
CERT="/opt/kubernetes/ssl/server.pem"
EKY="/opt/kubernetes/ssl/server-key.pem"
ENDPOINTS="192.168.1.36:2379"
ETCDCTL_API=3 etcdctl \
  --cacert="${CACERT}" --cert="${CERT}" --key="${EKY}" \
  --endpoints=${ENDPOINTS} \
  snapshot save /data/etcd_backup_dir/etcd-snapshot-`date +%Y%m%d`.db
# Keep backups for 30 days
find /data/etcd_backup_dir/ -name *.db -mtime +30 -exec rm -f {} \;

Restoring from a snapshot uses:

ETCDCTL_API=3 etcdctl snapshot restore /data/etcd_backup_dir/etcd-snapshot20191222.db \
  --name etcd-0 \
  --initial-cluster "etcd-0=https://192.168.1.36:2380,etcd1=https://192.168.1.37:2380,etcd-2=https://192.168.1.38:2380" \
  --initial-cluster-token etcd-cluster \
  --initial-advertise-peer-urls https://192.168.1.36:2380 \
  --data-dir=/var/lib/etcd/default.etcd

Resource‑object backup provides finer‑grained protection, allowing you to back up specific namespaces, deployments, or PVCs. Open‑source tools such as Velero , PX‑Backup , and Kasten support this capability.

Velero’s description: "An open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes." It offers three main functions: cluster backup & restore, resource migration, and replication for testing.

Installation can be performed via Helm, YAML manifests, or the CLI. After installation, Velero creates several CustomResourceDefinitions (CRDs) in the velero namespace.

Scheduled backups are created with commands such as:

# Create a backup every 6 hours
velero create schedule daily‑backup --schedule="@every 24h" --include-namespaces web
# Create a weekly backup that lives for 90 days
velero create schedule weekly‑backup --schedule="@every 168h" --ttl 2160h0m0s

Example output after creating a schedule shows the Schedule object in the velero namespace.

Cluster migration backup can be performed on demand with:

velero backup create test01 --include-namespaces default
# Describe or view logs
velero backup describe test01
velero backup logs test01

Restoring the backup in another cluster is as simple as:

velero restore create --from-backup test01

PVC backup and migration are supported for cloud‑native storage (EBS, Azure Disk, GCE PD) via snapshotting, or for other storage types through plugins. Adding the annotation backup.velero.io/backup-volumes: mypvc to a pod enables volume‑level backup.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    backup.velero.io/backup-volumes: mypvc
  name: rbd-test
spec:
  containers:
  - name: web-server
    image: nginx
    volumeMounts:
    - name: mypvc
      mountPath: /var/lib/www/html
  volumes:
  - name: mypvc
    persistentVolumeClaim:
      claimName: rbd-pvc-zhf
      readOnly: false

Backup creation for the PVC can be triggered with:

velero backup create testpvc05 --snapshot-volumes=true --include-namespaces default

Hooks allow execution of commands inside pods during backup. An example hook configuration freezes the filesystem before snapshotting:

metadata:
  name: nginx-deployment
  namespace: nginx-example
spec:
  template:
    metadata:
      annotations:
        pre.hook.backup.velero.io/container: fsfreeze
        pre.hook.backup.velero.io/command: '["/sbin/fsfreeze","--freeze","/var/log/nginx"]'
        post.hook.backup.velero.io/container: fsfreeze
        post.hook.backup.velero.io/command: '["/sbin/fsfreeze","--unfreeze","/var/log/nginx"]'

Other backup solutions mentioned include PX‑Backup (commercial), Kanister (focuses on data‑level snapshots), and various community links for further reading.

Reference links: Velero GitHub , Portworx , Kasten , Kanister , and several blog posts detailing backup implementations.