Deactivate all Ceph OSDs on a node.

Oct 24, 2021 2 minutes to read Wolf Noble Ceph Proxmox Tech Ceph Debian Howto Maintenance Proxmox Ubuntu WolfspyreLabs

Disabling Ceph OSDs on a node for maintenance

Oh wow this is a pain in the ass

So, I needed to do some infra maintenance. and quickly discovered that the process of safely offlining the OSDs on a node was taking forever.

Scriptification for the win

The steps we need to take are fairly simple.

The Steps to automate:

Set ceph to not freak out about offline OSDs ↕

ceph osd set noout Which gets us the response: noout is set

Enumerate all the local OSDs ↕

1
2
3
for osd in `ls -lA /var/lib/ceph/osd|awk '/ceph/{print $9}'|sed -e 's/ceph-//'`; do
  echo $osd;
done

Stop each OSD ↕

`1`	`service ceph-osd@OSD_NUMBER_HERE stop`

Clear to perform Maintenance ↕

Lets get to it, Shall we?

shutdown -h now

Do the needful ↕

perform maintenance

Bring it all back online ↕

press power button

Disable Noout ↕

1
ceph osd set noout

Tie it all together

1
2
3
4
5
6
7
8
9
for osd in `ls -lA /var/lib/ceph/osd|awk '/ceph/{print $9}'|sed -e 's/ceph-//'`; do
  echo $osd;
  service "ceph-osd@${osd}" stop >/dev/null 2>&1 &
  for i in `seq 1 3`; do
    echo -n '.' ;
    sleep 1;
  done;
  echo;
done

Do Maintenance here

Start node back up

Validate ceph is healthy

`1`	`/usr/bin/watch /usr/bin/ceph status`

Disable Noout

1
ceph osd set noout