Booting Proxmox from alternate disks

May 2, 2022 4 minutes to read Wolf Noble Howto Linux Proxmox Tech Boot Proxmox PXM Ubuntu WolfspyreLabs

Booting Proxmox from alternate disks

Pre-requisites

Install proxmox with debug mode enabled
After the install finishes, you’ll be dropped to a shell prompt.
This is where we begin our journey.

post-install tweaking

Re-import zfs pool

`1`	`zpool import -f -R /mnt rpool`

Bindmounts

1
2
3
4
5

for target in proc sys dev run; do
  echo "mounting ${target}"
  mount -o rbind "/${target}" "/mnt/${target}"
done

Chroot

`1`	`chroot /mnt /bin/bash`

Find the disks in question

1
fdisk -l |less

1
2
sfdisk -d /dev/sda > /tmp/starter
sfdisk -X gpt -T |egrep '(EFI|BIOS)' >> /tmp/starter

1
2
3
4
5
6
7
8
label: gpt
device: /dev/sdd
unit: sectors
sector-size: 512

size=2014, type=21686148-6449-6E6F-744E-656564454649, name="Boot"
size=1048576, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, name="EFIBoot"
size=10485760, type=L, name="Scratch"

Create GPT partition map on the disk(s)

`1`	`sfdisk /dev/sdd </tmp/starter`

Create partitions and filesystems

1
mkfs.ext4 -m 0 -L Scratch -M /scratch /dev/sdd3

create /scratch

1
2
mkdir /scratch
echo '/dev/disk/by-label/Scratch /scratch ext4 defaults 0 0' >> /etc/fstab

Fix boot

Add boot params to account for zfs

append “rootdelay=5” to /etc/kernel/cmdline

Now re-deploy the bootblock, kernel, initrd.

proxmox-boot-tool format /dev/sdd2

proxmox-boot-tool init /dev/sdd2

Delete the original bootloader

proxmox-boot-tool clean

Exit chroot

Unmount

1
2
3
4
for target in 'proc' 'sys/firmware/efi/efivars' 'sys' 'dev/pts' 'dev/shm' 'dev' 'run'; do
  echo "unmounting /mnt/${target}"
  umount "/mnt/${target}"
done

Reboot

Get confused.

Now… if you HADN’T added the rootdelay statement to the kernel cmdline above, proxmox won’t boot like you’d expect.

Previously I thought we needed to do this: zpool export rpool

But it seems like that… wasn’t my best idea, as on the subsequent boot I had to manually:

zpool import rpool to get things rolling for that boot…

However…. upon the next system boot this was still a problem. I found this: proxmox forum post which helped me realize we need to tell the kernel that it needs to wait for the block devices to be presented somehow.

Fix the Problem

Update the kernel commandline

we need to:

Append “rootdelay=10” to /etc/kernel/cmdline
Re-run pve-efiboot-tool refresh
Reboot

Troubleshooting

A problem we ran into while playing was that the kernel tries to boot before zfs has made things available. the addition of the rootdelay parameter to /etc/kernel/cmdline mentioned here was what fixed this for us.. You may need to increase that timeout, depending on your environment.

Other existing art

Click to see other examples found in the wild ...

[Mark Schouten][3] wrote a quick howto posted to the pve users mailing list:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
Hi,

one of my colleagues mistakenly installed a Proxmox node with LVM instead of ZFS, and I want to fix that without reinstalling. I tested the following steps, which seem to be working as it should. But maybe somebody can think of something that I forgot. So I thought I'd share it here.

Feel free to comment!

/dev/sdb is the unused device, /dev/sda is the currently in-use device.


***@proxmoxlvmzfs:~# apt install parted
***@proxmoxlvmzfs:~# parted -s /dev/sdb mktable gpt
***@proxmoxlvmzfs:~# parted -s /dev/sdb mkpart extended 34s 2047s
***@proxmoxlvmzfs:~# parted -s /dev/sdb mkpart extended 2048s 100%
***@proxmoxlvmzfs:~# parted -s /dev/sdb set 1 bios_grub on

***@proxmoxlvmzfs:~# zpool create -f rpool /dev/sdb2
***@proxmoxlvmzfs:~# zfs create rpool/ROOT
***@proxmoxlvmzfs:~# zfs create rpool/ROOT/pve-1
***@proxmoxlvmzfs:~# zfs create rpool/data
***@proxmoxlvmzfs:~# zfs create rpool/swap -V 8G
***@proxmoxlvmzfs:~# mkswap /dev/zvol/rpool/swap
***@proxmoxlvmzfs:~# cd /rpool/ROOT/pve-1
***@proxmoxlvmzfs:/rpool/ROOT/pve-1# rsync -avx / ./
***@proxmoxlvmzfs:/rpool/ROOT/pve-1# mount --bind /proc proc
***@proxmoxlvmzfs:/rpool/ROOT/pve-1# mount --bind /dev dev
***@proxmoxlvmzfs:/rpool/ROOT/pve-1# mount --bind /sys sys
***@proxmoxlvmzfs:/rpool/ROOT/pve-1# swapoff -a
***@proxmoxlvmzfs:/rpool/ROOT/pve-1# chroot .
================ fstab fix ================
Change swap partition to /dev/zvol/rpool/swap
Remove / mount entry
================ fstab fix ================
================ grub fix ================
In /etc/default/grub, set:
GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/pve-1 boot=zfs"
================ grub fix ================
***@proxmoxlvmzfs:/# zpool set bootfs=rpool/ROOT/pve-1 rpool
***@proxmoxlvmzfs:/# grub-install /dev/sda
***@proxmoxlvmzfs:/# grub-install /dev/sdb
***@proxmoxlvmzfs:/# update-grub
***@proxmoxlvmzfs:/# zfs set mountpoint=/ rpool/ROOT/pve-1

Reboot

***@proxmoxlvmzfs:~# lvchange -an pve
***@proxmoxlvmzfs:~# sgdisk /dev/sdb -R /dev/sda
***@proxmoxlvmzfs:~# sgdisk -G /dev/sda
***@proxmoxlvmzfs:~# zpool attach rpool /dev/sdb2 /dev/sda2