A while ago I started writing this so I figured instead of deleting it, might as well post it. Looks like the lesson learned is that fuse doesn’t give you all the speed you could possibly have.
In zfs-fuse I was getting 1.18M for the writes in iostat. When I disabled all caching, I got that up to 1.30M.
When I installed ubuntu-zfs, I was getting 3.88M with caching disabled. Turning back the cache to primarycache=metadata secondarycache=none drops the speed down to 3.22M.
I’ve realized since that I can use my old 250GB harddrive as a secondary cache device. To add it to my existing pool called zfs_4TB2, I just did:
sudo zpool add zfs_4TB2 cache /dev/sdb
In hindsight I probably should have used the ‘/dev/disk/by-id/…’ method, but oh well.
Setting the cache to primarycache=metadata secondarycache=all makes the speed 1.20M. After 6 hours of running, it used 9.5GB on the secondary cache according to
sudo zpool iostat -v