Kernel Integrated exFAT Support Performance has saved my Chia farmer

In case you did not hear, great exFAT support has been integrated in the Linux Kernel 5.7 quite some time ago. Unfortunately, that Kernel is not the default kernel for all Linux Distributions, namely Ubuntu. In this article we will see how drastic the improvements compared to the FUSE implementation are, and how to upgrade Ubuntu 20.04 to use the Kernel integrated exFAT drivers.

The result on my Chia farm is actually spectacular. Previously I have missed many challenges and the system was partially slow. Upgrading to the latest Kernel with integrated exFAT support, this changed completely. 

Before and after migrating to kernel integrated exFAT. Yes, Seconds!

Context

There have been multiple ways how to operate a exFAT filesystem with Linux. The most popular might still be the first one, as it is the default method in many major distros

• extfat Fuse relan/exfat: Free exFAT file system implementation (github.com)
• exfatNoFuse (with dkms): dorimanx/exfat-nofuse: Android ARM Linux non-fuse read/write kernel driver for exFat and VFat Android file systems (github.com)
• Mainline Kernel (since 5.7) with extFAT support.
• Backport: arter97/exfat-linux: exFAT filesystem module for Linux kernel (github.com)

I was using the "default" way, which has lead to quite some performance problems. A single drive has used up to 10% of my CPU core, just to provide read access and very low throttled write load (30MB/s). Having 4 USB drives connected to my mini PC, that meant 40% CPU usage, with little room for other CPU hungry services like SMB. After the migration, this dropped to below 10% for all 4 drives!

Interestingly is the availability of a backport of the Kernel integrated module. Their website list an interesting benchmark, which matches my experience

random IO on a NVMe

The second row is the Kernel Integrated version (5.7) compared to the backport (Bold). You see that the speed improvements are huge and almost on par with ext4. More benchmarks are available on their GitHub Repo

Some other real-life workloads, are also very positively affected, as this user mentions on Reddit: Anecdotal: Big performance difference in exfat-fuse vs exfat-nonfuse : archlinux (reddit.com)

How to Benefit?

Assuming that your running your extFat devices with Ubuntu 20.04, there are only a few steps required to bring the performance improvements to you. 

Install the newest Kernel

You can find multiple options for upgrading your kernel on Install or Upgrade To Latest Linux Kernel on Ubuntu 20.04 & Linux Mint 20 (linuxhint.com). For me, the following worked like a charm:

Steo 1: First you might check what is your current Kernel version:

mis@box:~$ uname -r 5.4.0-56-generic

This indicates that you're running kernel version 5.4

Step 2: The next steps is to download a script maintained by Roy to automatically download and install a new kernel. Don't forget to make the file executable and move it to the /usr/local/bin folder

mis@box:~$ wget https://raw.githubusercontent.com/pimlie/ubuntu-mainline-kernel.sh/master/ubuntu-mainline-kernel.sh mis@box:~$ chmod +x ubuntu-mainline-kernel.sh mis@box:~$ sudo mv ubuntu-mainline-kernel.sh /usr/local/bin/

This will make the file executable and place it in the right folder

Step 3: Start the installation. This is as simple as executing the below command with root permissions

mis@box:~$ sudo ubuntu-mainline-kernel.sh –i

This will prompt for a confirmation during the installation, but runs fully automated otherwise.

Step 4: Restart the system.

mis@box:~$ sudo restart

This will prompt for a confirmation during the installation, but runs fully automated otherwise.

Uninstall FUSE Version

Even though the new Kernel is installed, all drives mentioned in /etc/fstab will still be mounted with the FUSE driver if installed. We need to uninstall that as well.

Step 1: Unmount all drives that are currently using exFat as filesystem. Check your /etc/fstab file for that and unmount all of them with sudo umount /mnt/mointpoint

Step 2: Uninstall the old FUSE drivers. sudo apt remove exfat-fuse exfat-utils

Step 3: Remount all drives. This time the Kernel Integrated driver should take over.  sudo mount -a
You can check your success with the top command. There shouldn't be any process related to exFat.

Install User space Utilities

This is optional if you only want to mount drives, but it contains the tools to run a filesystem check, or create a fresh filesystem on a block device. Source: exfatprogs/exfatprogs: exFAT filesystem userspace utilities (github.com)

mis@box:~$ sudo apt install exfatprogs

Done!. You will again find the tools for file-system checks and initialization as before.

Disclaimer (Error 0x8007003B)

All the FUSE and Kernel implementation lock of truncate, which is a serious problem with SAMBA as Windows clients will timeout when transferring very large files to a share on a exFAT Mount,  for instance here: Support fast space allocation? · Issue #45 · relan/exfat (github.com) but the whole internet is full of timeout-errors related to that.

So if you plan to share a exFAT mount with Samba for Windows clients... Just don't. As soon as the file reaches a certain size (probably around 16GB), the copy process fill fail.

Due to lack of truncate, the file is filled with Zeros before copying starts

After approximately 20s, you will see an error message on Windows, even while SAMBA is still allocating the space for the file.... A quick Google search unveils potentially hundreds of users that have the same problem.

Infamous Error 0x8007003B: An unexpected network error ocurred

More of the same error

• Samba on Raspi copy large files gives error 0x8007003B solution so far: don't use ExFAT or FAT32 - Raspberry Pi Forums
• [ubuntu] Shared drive over Samba, error transferring large files (ubuntuforums.org)
• [OpenWrt Wiki] SMB / Samba share overview (Windows file sharing)
• DD-WRT Forum :: View topic - Copy big file to samba shared disk may cause router dies
• Can't copy large files (RPI + SMB + HDD) - Raspberry Pi Forums

You have been warned 😀