How to Safely Change Ceph MON IP Addresses During Server Migration

Background

Recently the company's servers were moved to an off‑site data center. The servers include a Ceph storage cluster, and the migration requires changing the IP addresses of all servers, including the MON nodes. The official Ceph documentation provides methods for changing MON IPs within the same network and for changing all MON IPs when the cluster moves to a different network.

Changing MON IPs Within the Same Network

The official guide calls this the correct method. Simply changing the MON IP is insufficient; you must add a new MON node with the desired IP, ensure it joins the quorum, then remove the old MON and update

ceph.conf

so clients and daemons know the new address.

Manually Adding a MON Node

Create a data directory for the new MON on the target node (choose your own

{mon-id}

).

<code>mkdir /var/lib/ceph/mon/ceph-{mon-id}</code>

Obtain the MON keyring and save it temporarily.

<code>ceph auth get mon. -o /tmp/ceph.mon.keyring</code>

Export the cluster's MON map to a temporary file.

<code>ceph mon getmap -o /tmp/monmap</code>

Format the MON data directory, specifying the monmap and keyring.

<code>ceph-mon -i {mon-id} --mkfs --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring</code>

Start the MON process. Use

--public-addr {ip:port}

to bind the desired IP.

<code>systemctl start ceph-mon@{mon-id}
ceph-mon -i {mon-id} --public-addr {ip:port}</code>

Removing a MON Node

Stop the MON service.

<code>systemctl stop ceph-mon@{mon-id}</code>

Remove the MON from the cluster.

<code>ceph mon remove {mon-id}</code>

Delete the MON entry from

ceph.conf

.

Changing All MON IPs (Different Network)

When the entire cluster moves to a new subnet (e.g., from 192.168.28.0/24 to 192.168.244.0/24), generate a new monmap with the updated IPs and inject it into each MON.

Export the current monmap (if the cluster is unreachable, run on a MON node and specify the

{mon-id}

).

<code>ceph mon getmap -o /tmp/monmap
ceph-mon -i node3 --extract-monmap /tmp/monmap</code>

The monmap looks like:

<code># monmaptool --print /tmp/monmap
monmaptool: monmap file /tmp/monmap
epoch 7
fsid 9c079a1f-6fc2-4c59-bd4d-e8bc232d33a4
... (list of monitors) ...</code>

Remove the existing monitors from the monmap.

<code># monmaptool --rm node1 --rm node2 --rm node3 /tmp/monmap
monmaptool: removing node1
monmaptool: removing node2
monmaptool: removing node3
monmaptool: writing epoch 1 to /tmp/monmap (0 monitors)</code>

Add the new monitors with the new IPs.

<code># monmaptool --add node1 192.168.244.44 --add node2 192.168.244.45 --add node3 192.168.244.46 /tmp/monmap
monmaptool: writing epoch 1 to /tmp/monmap (3 monitors)</code>

Stop each MON service and inject the new monmap.

<code>ceph-mon -i {mon-id} --inject-monmap /tmp/monmap</code>

Restart the MON services.

<code>systemctl start ceph-mon@{mon-id}</code>

After following the official steps, the client could not connect to the Ceph cluster even though MON services were listening and ports were open. Checking the MON status showed normal election. The issue was that

ceph.conf

still contained the old MON configuration. Updating the file and restarting the MONs restored client connectivity.

<code>ceph daemon mon.cloudmap mon_status</code>

Official documentation: https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/