I’m playing with ZFS on Linux using Debian jessie (and eventually stretch). I decided I really wanted to use ZFS for everything, including /boot/grub and swap space. In addition, I wanted to boot using UEFI. This is how I did it.

The official ZFS Root on Debian HOWTO is much more cautious and wants you to create an ext4 partition for /boot/grub and a separate partition for swap. Being the idiot^Wdaredevil I am, I decided to flout that advice and go all-in on ZFS. The rest of this article details the steps involved in making this happen. (You really want to read the ZFS Root on Debian guide first, and then come back here to see where I deviated. Also, standard disclaimers apply: I’m doing this on a laptop I don’t care about and won’t use for any production purpose. If following this guide creates a monster that eats your disk and calls your mother names, I take no responsibility. It worked for me, but might not work for you. Phew.)

Get Your Install Environment Ready

This part is going to smell a lot like the official guide. I’ll make a note of where I deviate from it, though.

  1. Download the rEFInd Boot Manager. You’ll use it to reboot into your new Debian install the first time.

  2. Download and boot a Live CD, such as debian-live-8.3.0-amd64-standard.iso. The official guide specifically requires using a Live CD and not just an installer CD, so I used the “standard” non-graphical Live CD. Use whatever Live CD you are comfortable with, as long as it’s Debian 8. (Of course, I’m doing all this with an “iso-hybrid” image that I’ve dd‘ed to a USB stick.)

  3. Once booted, download and install the zfsonlinux package. This simply adds the ZoL package repository to your APT sources. 

    $ sudo -i
# wget -q http://archive.zfsonlinux.org/debian/pool/main/z/zfsonlinux/zfsonlinux_6_all.deb
# dpkg -i zfsonlinux_6_all.deb
Selecting previously unselected package zfsonlinux.
(Reading database ... 44337 files and directories currently installed.)
Preparing to unpack zfsonlinux_6_all.deb ...
Unpacking zfsonlinux (6) ...
Setting up zfsonlinux (6) ...

  4. Install the ZFS packages in the Live CD environment. This will compile the SPL and ZFS dynamic kernel modules, which will take some time. 

    # apt-get update
Hit http://archive.zfsonlinux.org jessie InRelease
Hit http://archive.zfsonlinux.org jessie/main amd64 Packages
Hit http://security.debian.org jessie/updates InRelease
Ign http://archive.zfsonlinux.org jessie/main Translation-en_US
Ign http://archive.zfsonlinux.org jessie/main Translation-en
Hit http://security.debian.org jessie/updates/main Sources
Hit http://security.debian.org jessie/updates/main amd64 Packages
Ign http://http.debian.net jessie InRelease
Hit http://security.debian.org jessie/updates/main Translation-en
Hit http://http.debian.net jessie Release.gpg
Hit http://http.debian.net jessie Release
Hit http://http.debian.net jessie/main Sources
Hit http://http.debian.net jessie/main amd64 Packages
Hit http://http.debian.net jessie/main Translation-en
Reading package lists... Done
# apt-get install linux-image-amd64 debian-zfs
[...]
Loading new spl-0.6.5 DKMS files...
First Installation: checking all kernels...
Building only for 3.16.0-4-amd64
Building initial module for 3.16.0-4-amd64
Done.
[...]
Loading new zfs-0.6.5.2 DKMS files..
First Installation: checking all kernels...
Building only for 3.16.0-4-amd64
Building initial module for 3.16.0-4-amd64
Done.
[...]

  5. Load the kernel module and verify that it loaded properly: 

    # modprobe zfs
# dmesg | grep ZFS
[ 7446.660850] ZFS: Loaded module v0.6.5.2-2, ZFS pool version 5000, ZFS filesystem version 5

Partition Your Disks

I am performing these steps on a laptop with a single drive; as such my setup will not uncover any issues that may arise from using multiple drives in a mirror or RAIDZ configuration. (Just FYI.)

  1. I want everything on ZFS, including /boot/grub and my swap partition. So I will create one large zpool using the entire disk: 

    # zpool create -o ashift=12 -o altroot=/mnt -m none rpool /dev/disk/by-id/<disk_name>
# zfs set atime=off rpool
# zfs set relatime=on rpool
# zfs set compression=lz4 rpool
    zpool create will label the disk using GPT and actually create two partitions; partition 1 is the ZFS pool, and partition 9 is for the EFI System Partition (ESP). We’ll use this later. I also go ahead and disable atime, enable relatime, and enable lz4 compression on the root dataset so that these are inherited all the way down. I can always override them in a child dataset if necessary. “Wait, so handing zpool(1m) a whole disk still creates two partitions?”, you ask? Yes. Yes it does. So then why do I care so much about not creating a third or fourth partition for /boot/grub or swap? Principles, my friend. And blatant disregard for rules. Moving on…

  2. Use zfs(1m) to create a zvol for swap space: 

    # zfs create -o compression=off -V 4G rpool/swap
# mkswap -L swap /dev/zvol/rpool/swap
Setting up swapspace version 1, size = 4194300 KiB
LABEL=swap, UUID=354f7cba-fa33-45bb-97c7-7c6aa6109b22
    I’ll make sure to add an entry into /etc/fstab later, after that file gets created.

  3. Create the ROOT filesystem. (This is not the same as /; this will be the parent dataset for all future boot environments.) 

    # zfs create -o mountpoint=none rpool/ROOT

  4. Create the initial “boot environment” and mount rpool at /rpool: 

    # zfs create -o mountpoint=/ rpool/ROOT/debian-1
# zfs set mountpoint=/rpool rpool
    Setting the mountpoint for the rpool filesystem deviates from the ZFS on Debian guide, but seems in-line with what OpenIndiana does by default. I do this now instead of when I created the pool, because otherwise the rpool/ROOT/debian-1 filesystem cannot mount properly.

  5. Set the bootfs property on the pool to the boot environment you just created: 

    # zpool set bootfs=rpool/ROOT/debian-1 rpool

  6. Create any other filesystems you want. If the filesystems should be managed as a unit (i.e., all part of the same boot environment), then make sure to create them under rpool/ROOT/debian-1. (While there’s not a beadm(1m)-type command for Linux yet, I still want to set up the system up as if there were. That way if/when beadm gets ported, the system will be ready to handle it.) 

    # zfs create -o mountpoint=/home rpool/home
# zfs create -o mountpoint=/usr rpool/ROOT/debian-1/usr
# zfs create -o mountpoint=/var rpool/ROOT/debian-1/var
# zfs create -o mountpoint=/var/tmp -o setuid=off rpool/ROOT/debian-1/var/tmp
# zfs create -o mountpoint=/tmp -o setuid=off rpool/tmp
    The ZFS on Debian guide mentions setting exec=off on the two tmp filesystems they create. When I followed those instructions, I got errors during DKMS compilation, I believe, so I left it enabled.

  7. Finally, export the zpool. 

    # zpool export rpool

Once all the above steps are done, you should now have the following layout (you might want to check this before exporting the pool):

# zpool get all rpool
NAME   PROPERTY                    VALUE                       SOURCE
rpool  size                        238G                        -
rpool  capacity                    0%                          -
rpool  altroot                     -                           default
rpool  health                      ONLINE                      -
rpool  guid                        14661786247243457179        default
rpool  version                     -                           default
rpool  bootfs                      rpool/ROOT/debian-1         local
rpool  delegation                  on                          default
rpool  autoreplace                 off                         default
rpool  cachefile                   -                           default
rpool  failmode                    wait                        default
rpool  listsnapshots               off                         default
rpool  autoexpand                  off                         default
rpool  dedupditto                  0                           default
rpool  dedupratio                  1.00x                       -
rpool  free                        237G                        -
rpool  allocated                   1.12G                       -
rpool  readonly                    off                         -
rpool  ashift                      12                          local
rpool  comment                     -                           default
rpool  expandsize                  -                           -
rpool  freeing                     0                           default
rpool  fragmentation               0%                          -
rpool  leaked                      0                           default
rpool  feature@async_destroy       enabled                     local
rpool  feature@empty_bpobj         active                      local
rpool  feature@lz4_compress        active                      local
rpool  feature@spacemap_histogram  active                      local
rpool  feature@enabled_txg         active                      local
rpool  feature@hole_birth          active                      local
rpool  feature@extensible_dataset  enabled                     local
rpool  feature@embedded_data       active                      local
rpool  feature@bookmarks           enabled                     local
rpool  feature@filesystem_limits   enabled                     local
rpool  feature@large_blocks        enabled                     local

# zfs list -t all -o name,type,mountpoint,compress,exec,setuid,atime,relatime
NAME                         TYPE        MOUNTPOINT  COMPRESS  EXEC  SETUID  ATIME  RELATIME
rpool                        filesystem  /rpool           lz4    on      on    off        on
rpool/ROOT                   filesystem  none             lz4    on      on    off        on
rpool/ROOT/debian-1          filesystem  /                lz4    on      on    off        on
rpool/ROOT/debian-1/usr      filesystem  /usr             lz4    on      on    off        on
rpool/ROOT/debian-1/var      filesystem  /var             lz4    on      on    off        on
rpool/ROOT/debian-1/var/tmp  filesystem  /var/tmp         lz4   off     off    off        on
rpool/home                   filesystem  /home            lz4    on      on    off        on
rpool/swap                   volume      -                off     -       -      -         -
rpool/tmp                    filesystem  /tmp             lz4   off     off    off        on

Install Debian

  1. Reimport the pool and create the cache file: 

    # zpool import -d /dev/disk/by-id -R /mnt rpool
# mkdir -p /mnt/etc/zfs
# zpool set cachefile=/mnt/etc/zfs/zpool.cache rpool
    In case you’re wondering about the options passed to zpool import (and don’t have a copy of the zpool(1m) man page), the -d option tells zpool to search the /dev/disk/by-id directory for importable pools, and the -R option tells it to use /mnt as the altroot and set the cachefile property to none. The latter gives us the chance to create the directory for the cache file first, then set the option appropriately.

  2. Since I didn’t create a separate partition for /boot/grub, I can skip some things in the official guide and move directly to using debootstrap(8) to install the base Debian system. 

    # apt-get install debootstrap
# debootstrap --arch=amd64 jessie /mnt http://httpredir.debian.org/debian/
[...]
I: Base system installed successfully.

Configure Your New Debian System

  1. Create /etc/{hosts,hostname}. Substitute whatever you’re going to name your machine for “debzfs” in the command below. 

    # echo debzfs > /mnt/etc/hostname
# sed -i -Ee "s#(127.+)#\1 debzfs#" /mnt/etc/hosts
# cat /mnt/etc/hosts
127.0.0.1	localhost debzfs
::1		localhost ip6-localhost ip6-loopback
ff02::1		ip6-allnodes
ff02::2		ip6-allrouters

  2. Create /mnt/etc/fstab. It should be empty (except for a comment, maybe), but a line needs to be added for the swap device. Here I use printf(1) for terseness, but you can use your favorite editor to do this as well. 

    # printf "/dev/zvol/rpool/swap\tnone\t\tswap\tdefaults\t0 0\n" >> /mnt/etc/fstab
# cat /mnt/etc/fstab
/dev/zvol/rpool/swap	none		swap	defaults	0 0

    If you created a dataset for /tmp not under the boot environment (like I did), you might notice at some point that systemd won’t mount /tmp before things want to use it. If that is the case, you can attempt to try the workaround documented here, or you can do what I did and just set the mountpoint on rpool/tmp to legacy and add a line to fstab, like this: 

    # zfs set mountpoint=legacy rpool/tmp
# printf "rpool/tmp\t\t/tmp\t\tzfs\tdefaults\t0 0\n" >> /mnt/etc/fstab
# cat /mnt/etc/fstab
/dev/zvol/rpool/swap	none		swap	defaults	0 0
rpool/tmp		/tmp		zfs	defaults	0 0
    I’d much rather figure out how to tell systemd to mount /tmp earlier, but until then this method certainly works.

  3. Do some minor network configuration. The ZFS on Debian page just puts all interface configuration into /mnt/etc/network/interfaces, but here I’ve chosen to create a separate file per-interface in /mnt/etc/network/interfaces.d/. It’s your choice. 

    # ls /mnt/etc/network/interfaces.d/
eth0  lo
# cat /mnt/etc/network/interfaces.d/eth0
auto eth0
iface eth0 inet dhcp
# cat /mnt/etc/network/interfaces.d/lo
auto lo
iface lo inet loopback

  4. Bind-mount some useful filesystems into the new system. 

    # for f in dev dev/pts proc sys; do mount -v --bind {,/mnt}/$f; done
mount: /dev bound on /mnt/dev.
mount: /dev/pts bound on /mnt/dev/pts.
mount: /dev/proc bound on /mnt/proc.
mount: /dev/sys bound on /mnt/sys.

  5. Go ahead and copy the zfsonlinux package you downloaded earlier into /mnt/root so you don’t have to download it again, then chroot(1) into your new system. 

    # cp zfsonlinux_6_all.deb /mnt/root/
# chroot /mnt /bin/bash --login

  6. Now that you’re in the new environment, you have to setup ZoL all over again…. (It seems that there is a documentation error in the official guide when running the locale-gen command. Use what I’ve got below instead.) 

    # apt-get install locales
[...]
# sed -i -Ee 's/# (en_US.UTF+)/\1/' /etc/locale.gen
# locale-gen
Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.
# apt-get install lsb-release
[...]
# dpkg -i /root/zfsonlinux_6_all.deb
Selecting previous unselected package zfsonlinux.
(Reading database ... 11210 files and directories currently installed.)
Preparing to unpack zfsonlinux_6_all.deb ...
Unpacking zfsonlinux (6) ...
Setting up zfsonlinux (6) ...
# apt-get update
[...]

Okay, hold on. I’m going to stop here and deviate once again from the ZFS on Debian guide.

APT-Pin ZoL Packages

In the realm of “that thing that happened that one time”, the first time I tried this setup I got into a situation after updating to Debian’s testing channel (stretch) where the spl package from Debian updated before the spl and zfs packages from ZoL, which broke ZFS for a while until I could uninstall the Debian version and reinstall the ZoL version. Since then I’ve created an preferences file to pin ZoL packages at a higher priority than the Debian ones. (I’m sure this will never come back to bite me, ever.) If you want to do this, create the file /etc/apt/preferences.d/50-zol-packages with the following content. If you’d rather not mess with the Natural Order of Things, then feel free to skip this part.

# cat <<EOF > /etc/apt/preferences.d/50-zol-packages
> Package: *
> Pin: origin archive.zfsonlinux.org
> Pin-Priority: 900
> EOF
# apt-cache policy spl
spl:
  Installed: 0.6.5-1
  Candidate: 0.6.5-1
  Version table:
 *** 0.6.5-1 0
        900 http://archive.zfsonlinux.org/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status
# apt-cache policy grub2-common
grub2-common:
  Installed: 2.02-beta2.9-ZOL11-7aa9f6
  Candidate: 2.02-beta2.9-ZOL11-7aa9f6
  Version table:
 *** 2.02-beta2.9-ZOL11-7aa9f6 0
        900 http://archive.zfsonlinux.org/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status
     2.02~beta2-22+deb8u1 0
        500 http://httpredir.debian.org/debian/ jessie/main amd64 Packages

(Debian jessie doesn’t have an spl package, but stretch (testing) does, so I’m showing the output for grub2-common as well to show you the difference.)

Install ZoL (Again) and GRUB2 EFI Support

And now we continue on. Installing the debian-zfs metapackage will install the spl, spl-dkms, zfs-dkms, and zfsutils packages. Note that the DKMS modules will get compiled (again) at this time.

Once more I’ve deviated from the ZFS on Debian instructions. Instead of installing grub-pc, I’m installing the grub-efi package because I want to boot via UEFI. Exciting times.

# apt-get install linux-image-amd64 debian-zfs
[...]
# apt-get install grub2-common grub-efi zfs-initramfs
[...]
# apt-get dist-upgrade
[...]

Digression: Get Ready for UEFI Boot

Keep in mind that the Debian live-installer you booted from didn’t boot in (U)EFI mode, so configuring GRUB2 will only halfway work. What we need to do is reboot into UEFI mode so that grub-install can access and set EFI variables properly. However, before we do that we can finish out the installation process and prepare partition 9 to be the ESP. We need to install the dosfstools package to be able to format part9 as VFAT. Note that my disk is /dev/sda; yours might be different. You can also use the /dev/disk/by-id/identifier-part9 path instead if you want to stay consistent with the rest of this process.

# apt-get install dosfstools
# mkdir /boot/efi
# mkfs.vfat /dev/sda9
mkfs.fat 3.0.27 (2014-11-12)
# mount /dev/sda9 /boot/efi
# grub-probe -d /dev/sda9
fat
# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.16.0-4-amd64
Found initrd image: /boot/initrd.img-3.16.0-4-amd64
done
# grub-install -d /usr/lib/grub/x86_64-efi /dev/sda # Note: actually unsuccessful!
Installing for x86_64-efi platform.
efibootmgr: EFI variables are not supported on this system.
efibootmgr: EFI variables are not supported on this system.
Installation finished. No error reported
# find /boot/efi -type f
/boot/efi/EFI/debian/grubx64.efi

We’re going to want to mount the ESP to /boot/efi, so add a line to /etc/fstab for it. (If you want, you can mount it read-only instead, or not automatically mount it, etc.)

# printf "/dev/sda9\t\t/boot/efi\tvfat\tdefaults\t0 1\n" >> /etc/fstab
# cat /etc/fstab
/dev/zvol/rpool/swap	none		swap	defaults	0 0
rpool/tmp		/tmp		zfs	defaults	0 0
/dev/sda9		/boot/efi	vfat	defaults	0 1

At this point, you will not be able to boot your install without using rEFInd. We’ll fix that in a minute. Let’s go ahead and finish out the instructions in the official guide.

# passwd root
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
# umount /boot/efi
# exit
logout
# for f in dev/pts dev proc sys boot/efi; do umount /mnt/$f; done
# zfs umount -a
# zpool export rpool

If umount(8) complains about not being able to unmount /sys, find out what other random things got mounted there while you weren’t looking (mount|grep /mnt/sys) and unmount those first, then try to unmount /sys again.

Reboot into rEFInd

Download the rEFInd Boot Manager USB image (if you haven’t already) and dd(1) it to a USB stick. (If you have a second thumb drive, use that so that you still have the Debian Live environment on the first stick in case you need it.) Once you’ve done that, reboot your machine. Now would be a good time to make sure your “BIOS” is set to boot into UEFI mode. After verifying that, insert the rEFInd stick and tell your system to boot to it.

Even though grub-install failed to write to the EFI variables, it still put all of the boot files where they needed to go, so rEFInd should be able to identify your Debian installation. Select it from the boot manager screen (if it’s not already selected) and boot.

Log in as root (because everything worked swimmingly and it actually booted, right? Right? Whew!) and use grub-install to fix the UEFI variables it couldn’t set last time.

# grub-install /dev/sda
Installing for x86_64-efi platform.
Installation finished. No error reported.

Before doing anything else, let’s reboot without rEFInd and make sure that all the UEFI stuff is set properly and that your machine can boot under its own power. If anything goes wrong, go back over this article and make sure that you performed all the steps (and that none of them failed, horribly or otherwise). You should also consult the ZFS Root on Debian documentation as well.

If you rebooted and everything worked, you should be able to log back in as root and start installing whatever other things you want! Before you get too far down the road, though, it’d probably be a good idea to snapshot your boot environment just in case.

# zfs snapshot -r rpool/ROOT/debian-1@2016020401-pristine
# zfs list -rt all -o name,type,used,refer,mountpoint rpool/ROOT
NAME                                             TYPE         USED  REFER  MOUNTPOINT
rpool/ROOT                                       filesystem  1.12G    96K  none
rpool/ROOT/debian-1                              filesystem  1.12G   554M  /
rpool/ROOT/debian-1@2016020401-pristine          snapshot        0   554M  -
rpool/ROOT/debian-1/usr                          filesystem   337M   337M  /usr
rpool/ROOT/debian-1/usr@2016020401-pristine      snapshot        0   337M  -
rpool/ROOT/debian-1/var                          filesystem   252M   252M  /var
rpool/ROOT/debian-1/var@2016020401-pristine      snapshot        0   252M  -
rpool/ROOT/debian-1/var/tmp                      filesystem    96K    96K  /var/tmp
rpool/ROOT/debian-1/var/tmp@2016020401-pristine  snapshot        0    96K  -

Conclusion

And this is where I leave you, with Debian installed on a ZFS root and booting from UEFI. Where you go from here is up to you. Install a desktop environment or two, all the utilities you want, create yourself a regular user, and enjoy the benefits of using ZFS.