Kernel Integrated exFAT Support Performance has saved my Chia farmer

In case you did not hear, great exFAT support has been integrated in the Linux Kernel 5.7 quite some time ago. Unfortunately, that Kernel is not the default kernel for all Linux Distributions, namely Ubuntu. In this article we will see how drastic the improvements compared to the FUSE implementation are, and how to upgrade Ubuntu 20.04 to use the Kernel integrated exFAT drivers.

The result on my Chia farm is actually spectacular. Previously I have missed many challenges and the system was partially slow. Upgrading to the latest Kernel with integrated exFAT support, this changed completely. 

Before and after migrating to kernel integrated exFAT. Yes, Seconds!

Context

There have been multiple ways how to operate a exFAT filesystem with Linux. The most popular might still be the first one, as it is the default method in many major distros

I was using the "default" way, which has lead to quite some performance problems. A single drive has used up to 10% of my CPU core, just to provide read access and very low throttled write load (30MB/s). Having 4 USB drives connected to my mini PC, that meant 40% CPU usage, with little room for other CPU hungry services like SMB. After the migration, this dropped to below 10% for all 4 drives!

Interestingly is the availability of a backport of the Kernel integrated module. Their website list an interesting benchmark, which matches my experience

random IO on a NVMe

The second row is the Kernel Integrated version (5.7) compared to the backport (Bold). You see that the speed improvements are huge and almost on par with ext4. More benchmarks are available on their GitHub Repo

Some other real-life workloads, are also very positively affected, as this user mentions on Reddit: Anecdotal: Big performance difference in exfat-fuse vs exfat-nonfuse : archlinux (reddit.com)

How to Benefit?

Assuming that your running your extFat devices with Ubuntu 20.04, there are only a few steps required to bring the performance improvements to you. 

Install the newest Kernel

You can find multiple options for upgrading your kernel on Install or Upgrade To Latest Linux Kernel on Ubuntu 20.04 & Linux Mint 20 (linuxhint.com). For me, the following worked like a charm:

Steo 1: First you might check what is your current Kernel version:

mis@box:~$ uname -r 5.4.0-56-generic

This indicates that you're running kernel version 5.4

Step 2: The next steps is to download a script maintained by Roy to automatically download and install a new kernel. Don't forget to make the file executable and move it to the /usr/local/bin folder

mis@box:~$ wget https://raw.githubusercontent.com/pimlie/ubuntu-mainline-kernel.sh/master/ubuntu-mainline-kernel.sh mis@box:~$ chmod +x ubuntu-mainline-kernel.sh mis@box:~$ sudo mv ubuntu-mainline-kernel.sh /usr/local/bin/

This will make the file executable and place it in the right folder

Step 3: Start the installation. This is as simple as executing the below command with root permissions

mis@box:~$ sudo ubuntu-mainline-kernel.sh –i

This will prompt for a confirmation during the installation, but runs fully automated otherwise.

Step 4: Restart the system.

mis@box:~$ sudo restart

This will prompt for a confirmation during the installation, but runs fully automated otherwise.

Uninstall FUSE Version

Even though the new Kernel is installed, all drives mentioned in /etc/fstab will still be mounted with the FUSE driver if installed. We need to uninstall that as well.

Step 1: Unmount all drives that are currently using exFat as filesystem. Check your /etc/fstab file for that and unmount all of them with sudo umount /mnt/mointpoint

Step 2: Uninstall the old FUSE drivers. sudo apt remove exfat-fuse exfat-utils

Step 3: Remount all drives. This time the Kernel Integrated driver should take over.  sudo mount -a
You can check your success with the top command. There shouldn't be any process related to exFat.

Install User space Utilities

This is optional if you only want to mount drives, but it contains the tools to run a filesystem check, or create a fresh filesystem on a block device. Source: exfatprogs/exfatprogs: exFAT filesystem userspace utilities (github.com)

mis@box:~$ sudo apt install exfatprogs

Done!. You will again find the tools for file-system checks and initialization as before.

Disclaimer (Error 0x8007003B)

All the FUSE and Kernel implementation lock of truncate, which is a serious problem with SAMBA as Windows clients will timeout when transferring very large files to a share on a exFAT Mount,  for instance here: Support fast space allocation? · Issue #45 · relan/exfat (github.com) but the whole internet is full of timeout-errors related to that.

So if you plan to share a exFAT mount with Samba for Windows clients... Just don't. As soon as the file reaches a certain size (probably around 16GB), the copy process fill fail.

Due to lack of truncate, the file is filled with Zeros before copying starts


After approximately 20s, you will see an error message on Windows, even while SAMBA is still allocating the space for the file.... A quick Google search unveils potentially hundreds of users that have the same problem.

Error 0x8007003B: An unexpected network error ocurred
Infamous Error 0x8007003B: An unexpected network error ocurred

More of the same error

Comments

Popular posts from this blog

Migrating from Arduino IDE to Visual Studio Code to PlatformIO

How to prepare for SHN in Singapore?