Lately I’ve been very unhappy with latest FreeBSD causing reboots randomly during disk resilvering. I simply cannot tolerate random reboots of my fileserver. This fact combined with the migration of OpenZFS to the ZFS on Linux code base means it’s time for me to move from a FreeBSD based ZFS NAS to a Linux-based one.
Sadly there aren’t many options in this space yet. I wanted something where basic tasks were taken care of, like what FreeNAS does, but also supports ZFS. The solution I settled on was ProxMox, which is a hypervisor, but it also has ZFS support.
The biggest drawback of ProxMox vs FreeNAS is the GUI. There are some disk-related GUI options in ProxMox, but mostly it’s VM focused. Thus, I had to configure my required services via CLI.
Following are the settings I used when I configured my NAS to run ProxMox.
If you don’t want to pay for a proxmox license, change the PVE enterprise repository to the free version by modifying /etc/apt/sources.list.d/pve-enterprise.list to the following:
deb http://download.proxmox.com/debian/pve buster pve-no-subscription
Then run at apt update & apt upgrade.
Edit /etc/postfix/main.cf and tweak your mail server config as needed (relayhost). Restart postfix after editing:
systemctl restart postfix
Forward mail for root to your own email
Edit /etc/aliases and add an alias for root to forward to your desired e-mail address. Add this line:
Import the pool using the zpool import -f command (-f to force import despite having been active in a different system)
zpool import -f
By default they’re imported into the main root directory (/). If you want to have them go to /mnt, use the zfs set mountpoint command:
zfs set mountpoint=/mnt/
Install and configure zfs-zed
apt install zfs-zed
Modify /etc/zfs/zed.d/zed.rc and uncomment ZED_EMAIL_ADDR, ZED_EMAIL_PROG, and ZED_EMAIL_OPTS. Edit them to suit your needs (default values work fine, they just need to be uncommented.) Optionally uncomment ZED_NOTIFY_VERBOSE and change to 1 if you want more verbose notices like what FreeNAS does (scrub notifications, for example.)
After modifying /etc/zfs/zed.d/zed.rc, restart zed:
systemctl restart zfs-zed
By default ProxMox scrubs each of your datasets on the second Sunday of every month. This cron job is located in /etc/cron.d/zfsutils-linux. Modify to your liking.
Snapshot & Replication
There are many different snapshot & replication scripts out there. I landed on Sanoid. Thanks to SvennD for helping me grasp how to get it working.
Install sanoid :
#Install necessary packages
apt install debhelper libcapture-tiny-perl libconfig-inifiles-perl pv lzop mbuffer git
# Clone repo, build deb, install
git clone https://github.com/jimsalterjrs/sanoid.git cd sanoid
ln -s packages/debian .
dpkg-buildpackage -uc -us
apt install ../sanoid_*_all.deb
Edit /etc/sanoid/sanoid.conf with a backup and retention schedule for each of your datasets. Example taken from sanoid documentation:
use_template = production
use_template = production
recursive = yes
process_children_only = yes
hourly = 4
# templates below this line #
frequently = 0
hourly = 36
daily = 30
monthly = 3
yearly = 0
autosnap = yes
autoprune = yes
Once sanoid.conf is to your liking, create a cron job to launch sanoid every hour (sanoid determines whether any action is needed when executed.)
#Add this line, save and exit
0 * * * * /usr/sbin/sanoid --cron
syncoid (part of sanoid) easily replicates snapshots. The syntax is pretty straightforward:
syncoid <source> <destination> -r
#-r means recursive and is optional
For remote locations specify a username@ before the ip/hostname, then a colon and the dataset name, for example:
syncoid email@example.com:sourceDataset localDataset -r
You can even have a remote source go to a different remote destination, which is pretty neat.
Other syncoid options of interest:
--debug #for seeing everything happening, useful for logging
--exclude #Regular expression to exclude certain datasets
--src-bwlimit #Set an upload limit so you don't saturate your bandwidth
--quiet #don't output anything unless it's an error
Automate synchronization by placing the same syncoid command into a cronjob:
0 */4 * * * /usr/sbin/syncoid --exclude=bigdataset1 --source-bwlimit=1M --recursive pool/data firstname.lastname@example.org:pool/data
#if you don't want status emails when the cron job runs, add --quiet
Install the nfs-kernel-server package and specify your NFS exports in /etc/exports.
apt install nfs-kernel-server portmap
Example /etc/exports :
Restart nfs-server after modifying your exports:
systemctl restart nfs-server
Install samba, configure /etc/samba/smb.conf, and add users.
apt install samba
systemctl enable smbd
/etc/samba/smb.conf syntax is fairly straightforward. See the samba documentation for more information. Example share configuration:
comment = Example share
path = /mnt/example
valid users = user1 user2
writable = yes
Add users to the system itself with the adduser command:
Add those same users to samba with the smbpasswd -a command. Example:
smbpasswd -a user1
Restart samba after making changes:
systemctl restart smbd
Taken from https://pve.proxmox.com/wiki/Disk_Health_Monitoring:
By default, smartmontools daemon smartd is active and enabled, and scans the disks under /dev/sdX and /dev/hdX every 30 minutes for errors and warnings, and sends an e-mail to root if it detects a problem.
Edit the file /etc/smartd.conf to suit your needs. You can specify/exclude devices, smart attributes, etc there. See here for more information. Restart the smartd service after modifying.
apc-upsd was easiest for me to configure, so I went with it. Thanks to this blog for giving me the information to get started.
First, install apcupsd:
apt install apcupsd apcupsd-doc
As soon as it was installed my console kept getting spammed about IRQ issues. To stop these errors I stopped the apcupsd daemon:
systemctl stop apcupsd
Now modify /etc/apcupsd/apcupssd.conf to suit your needs. The section I added for my CyberPower OR2200LCDRT2U was simply:
Then modify /etc/default/apcupsd to specify it’s configured:
After configuring, you can restart the apcupsd service
systemctl start apcupsd
To check the status of your UPS, you can run the apcaccess status command:
Install Logwatch to monitor system events. Here is a good primer on all of Logwatch’s options.
apt install logwatch
Modify /usr/share/logwatch/default.conf/logwatch.conf to suit your needs. By default it runs daily (defined in /etc/cron.daily/00logwatch). I added the following lines for my config to filter out unwanted information:
Service = "-zz-disk_space"
Service = "-postfix"
Service = "vsmartd"
Service = "-zz-lm_sensors"
Manually run logwatch to get a preview of what you’ll see:
logwatch --range today --mailto
ZFS-ZED not sending email
If ZED isn’t sending emails it’s likely due to an error in the config. For some reason default values still need to be uncommented for zed to work, even if left unaltered. Thanks to this post for the info.
Samba share access denied
If you get access denied when trying to write to a SMB share, double check the file permissions on the server level. Execute chmod / chown as appropriate. Example:
chown user1 -R /mnt/example/user1