My notes on changing which network your Proxmox CEPH cluster lives in. In my case I wanted to switch from a 10 gig network to a 40gig network in a different subnet. Source: https://forum.proxmox.com/threads/ceph-changing-public-network.119116

1. Change network configuration in “ceph.conf”
   • Be sure to edit both cluster network and public network
2. Destroy and recreate monitors (one by one);
3. Destroy and recreate managers (one by one, leaving the active one for last);
4. Destroy and recreate metadata servers (one by one, leaving the active one for last;
5. Restart OSDs (one by one – or more, depending how many OSDs you have in the cluster – so you avoid restarting the hosts);

Get CEPH running on new Proxmox node

pveceph install –repository no-subscription

Move OSDs to new host

Source: https://forum.proxmox.com/threads/move-osd-to-another-node.33965/page-2

Follow a similar procedure above of downing each OSD one by one on the old host. Remove the drives and place them in the new host. Then run the following:

pvscan
ceph-volume lvm activate --all

Troubleshooting

Unable to remove monitor with unknown status

https://forum.proxmox.com/threads/ceph-cant-remove-monitor-with-unknown-status.63613

rm -r /var/lib/ceph/mon/ceph-pve2/

Remove failed host

I had to edit /etc/pve/ceph.conf manually, remove host when it failed. It wouldn’t work in the Proxmox GUI.