Lately I’ve been very unhappy with latest FreeBSD causing reboots randomly during disk resilvering. I simply cannot tolerate random reboots of my fileserver. This fact combined with the migration of OpenZFS to the ZFS on Linux code base means it’s time for me to move from a FreeBSD based ZFS NAS to a Linux-based one.
Sadly there aren’t many options in this space yet. I wanted something where basic tasks were taken care of, like what FreeNAS does, but also supports ZFS. The solution I settled on was ProxMox, which is a hypervisor, but it also has ZFS support.
The biggest drawback of ProxMox vs FreeNAS is the GUI. There are some disk-related GUI options in ProxMox, but mostly it’s VM focused. Thus, I had to configure my required services via CLI.
Following are the settings I used when I configured my NAS to run ProxMox.
Repo setup
If you don’t want to pay for a proxmox license, change the PVE enterprise repository to the free version by modifying /etc/apt/sources.list.d/pve-enterprise.list to the following:
deb http://download.proxmox.com/debian/pve buster pve-no-subscription
Then run at apt update & apt upgrade.
Email alerts
Postfix configuration
Edit /etc/postfix/main.cf and tweak your mail server config as needed (relayhost). Restart postfix after editing:
systemctl restart postfix
Forward mail for root to your own email
Edit /etc/aliases and add an alias for root to forward to your desired e-mail address. Add this line:
root: YOUR_EMAIL_ADDRESS
Afterward run:
newaliases
ZFS configuration
Pool Import
Import the pool using the zpool import -f command (-f to force import despite having been active in a different system)
zpool import -f
By default they’re imported into the main root directory (/). If you want to have them go to /mnt, use the zfs set mountpoint command:
zfs set mountpoint=/mnt/
Monitoring
Install and configure zfs-zed
apt install zfs-zed
Modify /etc/zfs/zed.d/zed.rc and uncomment ZED_EMAIL_ADDR, ZED_EMAIL_PROG, and ZED_EMAIL_OPTS. Edit them to suit your needs (default values work fine, they just need to be uncommented.) Optionally uncomment ZED_NOTIFY_VERBOSE and change to 1 if you want more verbose notices like what FreeNAS does (scrub notifications, for example.)
After modifying /etc/zfs/zed.d/zed.rc, restart zed:
systemctl restart zfs-zed
Scrubbing
By default ProxMox scrubs each of your datasets on the second Sunday of every month. This cron job is located in /etc/cron.d/zfsutils-linux. Modify to your liking.
Snapshot & Replication
There are many different snapshot & replication scripts out there. I landed on Sanoid. Thanks to SvennD for helping me grasp how to get it working.
#Install necessary packages apt install debhelper libcapture-tiny-perl libconfig-inifiles-perl pv lzop mbuffer git # Clone repo, build deb, install git clone https://github.com/jimsalterjrs/sanoid.git cd sanoid ln -s packages/debian . dpkg-buildpackage -uc -us apt install ../sanoid_*_all.deb
Snapshots
Edit /etc/sanoid/sanoid.conf with a backup and retention schedule for each of your datasets. Example taken from sanoid documentation:
[data/home]
use_template = production
[data/images]
use_template = production
recursive = yes
process_children_only = yes
[data/images/win7]
hourly = 4
#############################
# templates below this line #
#############################
[template_production]
frequently = 0
hourly = 36
daily = 30
monthly = 3
yearly = 0
autosnap = yes
autoprune = yes
Once sanoid.conf is to your liking, create a cron job to launch sanoid every hour (sanoid determines whether any action is needed when executed.)
crontab -e #Add this line, save and exit 0 * * * * /usr/sbin/sanoid --cron
Replication
syncoid (part of sanoid) easily replicates snapshots. The syntax is pretty straightforward:
syncoid <source> <destination> -r #-r means recursive and is optional
For remote locations specify a username@ before the ip/hostname, then a colon and the dataset name, for example:
syncoid root@10.0.0.1:sourceDataset localDataset -r
You can even have a remote source go to a different remote destination, which is pretty neat.
Other syncoid options of interest:
--debug #for seeing everything happening, useful for logging
--exclude #Regular expression to exclude certain datasets
--src-bwlimit #Set an upload limit so you don't saturate your bandwidth
--quiet #don't output anything unless it's an error
Automate synchronization by placing the same syncoid command into a cronjob:
0 */4 * * * /usr/sbin/syncoid --exclude=bigdataset1 --source-bwlimit=1M --recursive pool/data root@192.168.1.100:pool/data #if you don't want status emails when the cron job runs, add --quiet
NFS
Install the nfs-kernel-server package and specify your NFS exports in /etc/exports.
apt install nfs-kernel-server portmap
Example /etc/exports :
/mnt/example/DIR1 192.168.0.0/16(rw,sync,all_squash,anonuid=0,anongid=0)
Restart nfs-server after modifying your exports:
systemctl restart nfs-server
Samba
Install samba, configure /etc/samba/smb.conf, and add users.
apt install samba systemctl enable smbd
/etc/samba/smb.conf syntax is fairly straightforward. See the samba documentation for more information. Example share configuration:
[exampleshare] comment = Example share path = /mnt/example valid users = user1 user2 writable = yes
Add users to the system itself with the adduser command:
adduser user1
Add those same users to samba with the smbpasswd -a command. Example:
smbpasswd -a user1
Restart samba after making changes:
systemctl restart smbd
SMART monitoring
Taken from https://pve.proxmox.com/wiki/Disk_Health_Monitoring:
By default, smartmontools daemon smartd is active and enabled, and scans the disks under /dev/sdX and /dev/hdX every 30 minutes for errors and warnings, and sends an e-mail to root if it detects a problem.
Edit the file /etc/smartd.conf to suit your needs. You can specify/exclude devices, smart attributes, etc there. See here for more information. Restart the smartd service after modifying.
UPS monitoring
apc-upsd was easiest for me to configure, so I went with it. Thanks to this blog for giving me the information to get started.
First, install apcupsd:
apt install apcupsd apcupsd-doc
As soon as it was installed my console kept getting spammed about IRQ issues. To stop these errors I stopped the apcupsd daemon:
systemctl stop apcupsd
Now modify /etc/apcupsd/apcupssd.conf to suit your needs. The section I added for my CyberPower OR2200LCDRT2U was simply:
UPSTYPE usb
DEVICE
Then modify /etc/default/apcupsd to specify it’s configured:
#/etc/default/apcupsd ISCONFIGURED=yes
After configuring, you can restart the apcupsd service
systemctl start apcupsd
To check the status of your UPS, you can run the apcaccess status command:
/sbin/apcaccess status
Log monitoring
Install Logwatch to monitor system events. Here is a good primer on all of Logwatch’s options.
apt install logwatch
Modify /usr/share/logwatch/default.conf/logwatch.conf to suit your needs. By default it runs daily (defined in /etc/cron.daily/00logwatch). I added the following lines for my config to filter out unwanted information:
Service = "-zz-disk_space" Service = "-postfix" Service = "vsmartd" Service = "-zz-lm_sensors"
Manually run logwatch to get a preview of what you’ll see:
logwatch --range today --mailto YOUR_EMAIL_ADDRESS
UPDATE 8/29/2020
I discovered additional tweaking to logwatch to get it exactly how I like it (thanks to this post and this one at serverfault.)
Defaults for monitored services are located in /usr/share/logwatch/default.conf/services/
You can copy this default file to /etc/logwatch/conf/services/<filename.conf> and then modify the service as needed. In my case I wanted to ignore logins for a particular user from a particular machine. This can be done by copying & editing sshd.conf and adding the following:
# Ignore these hosts
*Remove = 192.168.100.1
*Remove = X.Y.123.123
# Ignore these usernames
*Remove = testuser
# Ignore other noise. Note that we need to escape the ()
*Remove = "pam_succeed_if\(sshd:auth\): error retrieving information about user netscan.*Troubleshooting
ZFS-ZED not sending email
If ZED isn’t sending emails it’s likely due to an error in the config. For some reason default values still need to be uncommented for zed to work, even if left unaltered. Thanks to this post for the info.
Samba share access denied
If you get access denied when trying to write to a SMB share, double check the file permissions on the server level. Execute chmod / chown as appropriate. Example:
chown user1 -R /mnt/example/user1