FreeBSD 8.0 ZFS Tuning

I recently migrated my RAID-5 home storage array to RAIDZ using ZFS on FreeBSD 8.0-RELEASE. FreeBSD’s latest release includes ZFS v13 with many tuning patches, making ZFS not only more stable but easier to tune.

It is noted that on 64-bit systems with over 2GB RAM, default auto-tuning settings will be quite stable. However in any other configuration some tuning is still necessary to achieve a stable system.

On 32-bit you still have to compile a custom kernel per these instructions. If it is your first time compiling a custom kernel, don’t worry it is a very short procedure. Kernel KVA_PAGES (virtual address space) must be increased to allow for ZFS to use more kernel memory. The setting for this value depends on other userland applications that will be running on the server, and how much memory they require. KVA_PAGES value must be a multiple of 4 and that is how you can determine the kernel memory allocation, in my case:

KVA_PAGES=384 -> 384 * 4 -> 1536MB (1.5GB)

System configuration:

FreeBSD/i386 8.0 (Pentium 4)
2GB RAM
965GB ZFS RAID-Z over 3 drives

The general guideline I followed for setting tuning values was:

vm.kmem_size -> 1/2 of total RAM
vm.kmem_size_max -> 1/2 of total RAM
vfs.zfs.arc_max -> 1/2 of vm.kmem_size_max

However when testing at high transfer load (1000mbit LAN) with arc_max set to 512MB I still experienced kernel panics. I still suggest starting at 1/2 of kmem_size and working down from there.

Final stable tuning values:

# /boot/loader.conf
vm.kmem_size="1024M"
vm.kmem_size_max="1024M"
vfs.zfs.arc_max="256M"

# /etc/sysctl.conf
kern.maxvnodes="400000"

# Custom Kernel Options
options    KVA_PAGES=384    # increase kernel virtual address space for ZFS