Category Archives: OS

OS, Web

Nextcloud AIO mount to local directory reboot persistence

Leave a comment

I recently reconfigured my nextcloud AIO docker instance to get files from a local bind mount instead of a network share. I needed to mount the files with the www-data user (id 33) so nextcloud would properly read and write to them. The procedure of making a fuse mount with force-user and force-group as outlined in the readme did the trick.

However, it did not survive a reboot. The directory I want to expose to the nextcloud container is a ZFS dataset. There is some sort of race condition where the fuse filesystem sometimes binds to the empty mount directory before the dataset is mounted. To fix this, I needed to make the mount process its own systemd unit file, instead of an fstab entry. This is how I got it to work:

cat /etc/systemd/system/nextcloud-mount.service 
[Unit]
Description=bindfs mirror /mnt/storage -> /mnt/nextcloud_storage
RequiresMountsFor=/mnt/storage
After=local-fs.target remote-fs.target

[Service]
Type=forking
ExecStart=/usr/bin/bindfs -u 33 -g 33 /mnt/storage /mnt/nextcloud_storage
ExecStop=/bin/umount -l /mnt/nextcloud_storage
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

After writing that file, doing systemctl daemon-reload and systemctl enable nextcloud-mount my problem was solved! The bind mount only happens after all local and remote filesystems are mounted.

CLI, OS

CRON: Get full output only on error

Leave a comment

I found this very handy trick to cron jobs to make sure they only e-mail you on failure, and when they do, you get the entire output of the cronjob. Thanks to this serverfault.com post

The trick is to put the entire thing including STDOUT and STDERR in a subshell outputting to variable OUTPUT, and then append || echo “$OUTPUT” to the end of the command. If the command is successful, the echo half never fires, meaning silent execution. If it does error, then the full output is echoed, which will get picked up and e-mailed from cron (if configured correctly.)

The example I used is for paperless-ngx backups: 

OUTPUT=$(cd /mnt/docker/paperless && docker compose exec -T webserver document_exporter -p ../export/backup --delete 2>&1) || echo "$OUTPUT"

It works!

Networking, OS

NPM reboot workaround script

Leave a comment

I had an annoying “chicken and egg” problem with nginx proxy manager. It has hosts configurations that reference DNS names that are only reachable over VPN. It also hosts headscale, which is required for VPN to establish properly for said DNS entries. Brought up from scratch (say, after a reboot) NPM fails to launch because it can’t resolve the DNS entries. Headscale therefore isn’t reachable by the other nodes, and we’re stuck.

I developed a script to be run at boot to get around this annoying behavior of nginx proxy manager. It disables all other hosts in NPM except headscale, which doesn’t need DNS resolution as it resides on the same host. It then fires up NPM, and then tailscale, and waits to confirm the VPN is established and DNS resolution is working. Then it restores all the other hosts and restarts NPM. This script was 100% done myself, no AI assistance was used.

#!/bin/bash
# Script to properly bring things up after a reboot
# Name resolution doesn't happen until vpn VIP comes up
# NPM doesn't spin up until name resolution works
# vpn headscale is part of NPM
#
# Replace all VPN DNS host entries with 127.0.0.1
# Wait for NPM to come up
# Restore original host entries to get name resolution to work
# restart npm
#
# Run on a cron job @reboot. Ensure cron service is enabled.
# @reboot /docker/npm/reboot-script.sh | tee /tmp/reboot-script.log

# Variables
DOCKER_DIR=/docker/npm
CONF_DIR=$DOCKER_DIR/data/nginx/proxy_host
TEMP_DIR=/tmp/$(basename $CONF_DIR)
VPN_IP=100.1.1.1

# Log start of script
echo "Reboot script started on $(date)"

# Backup host entries
rsync -aP --delete $CONF_DIR $(dirname $TEMP_DIR) 

# Disable all hosts except for headscale
sed -i  's/\".*.<VPN_DNS_SUFFIX>\"/\"127.0.0.1\"/g' $CONF_DIR/*.conf
sed -i  's/http:\/\/.*.<VPN_DNS_SUFFIX>/http:\/\/127.0.0.1/g' $CONF_DIR/*.conf

# Listen on 127.0.0.1 for npm
sed -i "s/$VPN_IP/127.0.0.1/g" $DOCKER_DIR/docker-compose.yml

# Stop and restart npm
cd $CONF_DIR && docker compose down && docker compose up -d

# Wait for NPM to launch
until netstat -an|grep 0.0.0.0:80 >/dev/null; do echo "Waiting for NPM"; sleep 5; done; echo "NPM up"

# Restart tailscale
systemctl restart tailscaled

# Wait until VPN pings are successful
until ping -w 1 -c 1 <VPN_DNS_IP> >/dev/null; do echo "Waiting for VPN"; sleep 5; done; echo "VPN Successful"

# Restore config files
rsync -aP $TEMP_DIR/ $CONF_DIR
sed -i "s/127.0.0.1/$VPN_IP/g" $DOCKER_DIR/docker-compose.yml

# Restart npm
cd $CONF_DIR && docker compose down && docker compose up -d

echo "Reboot script completed on $(date)"

OS, Virtualization

Fix frozen boot / no network after upgrading to Proxmox 9

Leave a comment

I recently upgraded to Proxmox Virtual Environment 9 and was dismayed to see the server just hung forever after saying disks were initialized. A boot in debug mode (advanced grub boot option) revealed that systemd was hanging forever on waiting for the network service to start.

My first trick was to give the networking service a timeout by modifying /etc/systemd/system/network-online.target.wants/networking.service and adding TimeoutStartSec=90sec to the [Service] section. This at least allowed the system to boot normally and not hang anymore.

After boot, however, networking did not work. After much troubleshooting I found this article explaining it has to do with a post-up command I had for my mesh networking interface: post-up /usr/bin/systemctl restart frr.service

Once I removed that line, I was able to boot normally.

CLI, Hardware, OS, Virtualization

Add NVIDIA GPU to LXC container

Leave a comment

I followed this guide to get NVIDIA drivers working on my Proxmox machine. However when I tried to get them working in my container I couldn’t see how to get nvidia-smi installed. Thankfully this blog had what I needed.

The step I missed was copying & installing the NVIDIA drivers into the container with this flag:

--no-kernel-module

That got me one step closer but I could not spin up open-webui in a container. I kept getting the error 

Error response from daemon: could not select device driver "nvidia" with capabilities: [[gpu]]

The fix was to install the NVDIA Container Toolkit:

Configure the production repository:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Update the packages list from the repository:

sudo apt-get update

Install the NVIDIA Container Toolkit packages:

sudo apt-get install -y nvidia-container-toolkit

An additional hurtle I encountered was this error:

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: mount error: failed to add device rules: unable to find any existing device filters attached to the cgroup: bpf_prog_query(BPF_CGROUP_DEVICE) failed: operation not permitted: unknown

I found here that the fix is to change a line in /etc/nvidia-container-runtime/config.toml. Uncomment and change no-cgroups to true.

no-cgroups = true

Success.

Not working after reboot

I had a working config until I rebooted the host. It turns out that two services need to run on the host:

nvidia-persistenced
nvidia-smi

Configured cron tab to run these on reboot:

/etc/cron.d/nvidia:
@reboot root /usr/bin/nvidia-smi
@reboot root /usr/bin/nvidia-persistenced

Update 2025-05-06

I encountered an error when trying to set up alltalk tts:


nvidia-container-cli: mount error: stat failed: /dev/nvidia-modeset: no such file or directory: unknown

It turns out I needed to expose /dev/nvidia-modeset to the container as well. Thanks to this reddit post for the answer. The complete container passthrough config is now this:

lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 243:* rwm
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file

Update 2026-09-30

New error this time: invalid device node: /dev/nvidia-modeset

Fixed thanks to https://github.com/NVIDIA/nvidia-container-toolkit/issues/139

Add this line to your container config file and restart your container:

 lxc.mount.entry: /proc/driver/nvidia/gpus/0000:01:00.0 proc/driver/nvidia/gpus/0000:01:00.0 none bind,optional,create=file
CLI, OS

Change ceph network

Leave a comment

My notes on changing which network your Proxmox CEPH cluster lives in. In my case I wanted to switch from a 10 gig network to a 40gig network in a different subnet. Source: https://forum.proxmox.com/threads/ceph-changing-public-network.119116

  1. Change network configuration in “ceph.conf”
    • Be sure to edit both cluster network and public network
  2. Destroy and recreate monitors (one by one);
  3. Destroy and recreate managers (one by one, leaving the active one for last);
  4. Destroy and recreate metadata servers (one by one, leaving the active one for last;
  5. Restart OSDs (one by one – or more, depending how many OSDs you have in the cluster – so you avoid restarting the hosts);

Get CEPH running on new Proxmox node

pveceph install –repository no-subscription

Move OSDs to new host

Source: https://forum.proxmox.com/threads/move-osd-to-another-node.33965/page-2

Follow a similar procedure above of downing each OSD one by one on the old host. Remove the drives and place them in the new host. Then run the following:

pvscan
ceph-volume lvm activate --all

Troubleshooting

Unable to remove monitor with unknown status

https://forum.proxmox.com/threads/ceph-cant-remove-monitor-with-unknown-status.63613

rm -r /var/lib/ceph/mon/ceph-pve2/

Remove failed host

I had to edit /etc/pve/ceph.conf manually, remove host when it failed. It wouldn’t work in the Proxmox GUI.

Hardware, OS

Docker for old Java-based IPMI systems

Leave a comment

As time goes on it’s getting harder to access the remote console on one of my older systems. It requires an old version of Java that just doesn’t work right on modern operating systems. Fortunately, Docker has come to my rescue.

ipmi-kvm-docker is a simple docker setup that will spin up an old version of firefox with the old version of Java that’s required. It wraps it all nicely in a graphical environment inside the docker container and uses novnc to serve this environment up as a web page. You can configure which port it uses, and then you simply connect to your docker host on that port. You can also specify a volume to mount if you have ISOs or other files you’d like your IPMI environment to see. Brilliant.

I simply ran these commands on my docker system to get it up and running:

docker run -p 8080:8080 -v /your/local/folder:/root/images solarkennedy/ipmi-kvm-docker

Then all I had to do was connect to my docker system on port 8080 in a browser. It worked great. My 10+year old server is still going strong, and easy to reinstall now thanks to this tool.

CLI, OS

Add multiple search domains in CentOS 7

Leave a comment

I needed to add multiple domains to search DNS with on my Cent7 box. It turns out there are two ways to do it. Cent7 uses networkmanager, so you can use the cli tool to add what you want, or you can edit the file directly.

Using nmcli:

sudo nmcli con mod eth0 ipv4.dns-search "domain1.org,domain2.org,domain3.org"

This causes nmcli to add this line to your network interface config file (/etc/sysconfig/network-scripts/ifcfg-eth0 in my case)

DOMAIN="domain1.org domain2.org domain3.org"

After either using nmcli or manually editing your file, simply restart the network service and your search domains now work!

CLI, OS

Restore files from remote borg repository disk image

Leave a comment

My off-site backup involves sending borgbackup archives of VM images to a remote synology server. I recently needed to restore a single file from one of the VM images stored within this borg backup repository on the remote server. My connection to this server is not very fast so I didn’t want to wait to download the entire image file to mount it locally.

My solution was to mount the remote borgbackup repository on my local machine over SSH so I could poke around for and copy the specific file I wanted. This requires the borgbackup binary to be present on the remote machine. Since it’s a synology, I simply copied the standalone binary over.

The restore process was complicated by the fact that the VM disk image is owned by root, so in order to access the file I needed to mount the remote repository as root.

This is the process:

  1. Set BORG_REMOTE_PATH
    1. export BORG_REMOTE_PATH=<PATH_TO_BORG_BINARY_ON_REMOTE_SYSTEM>
  2. (Arch Linux): install python-llfuse
  3. Mount repository over SSH:
    1. borg mount <USER>@<REMOTE_SYSTEM>:<PATH_TO_REMOTE_BORGBACKUP_REPOSITORY>::<BACKUP_NAME> <MOUNT_FOLDER>
  4. Follow disk image mounting process
    1. losetup -Pr -f <PATH_TO_MOUNTED_BORGBACKUP>/<FILENAME_OF_VM_IMAGE>
    2. mount -o ro /dev/loop0p2 /mnt/loop0/
  5. Follow reverse to unmount when done:
    1. umount /mnt/loop0
    2. losetup -d /dev/loop0
    3. borg umount <MOUNT_FOLDER>

Success! I was able to restore an individual file within a raw VM image backup on a remote Borgbackup repository using this method.

CLI, Networking, OS

send test syslog messages with nc

Leave a comment

I needed to send some test packets over UDP to make sure connectivity was working. I found this site which outlined how to do it really well

nc -u <IP/hostname> <port>

Then on the next line you can send test messages, then hit CTRL+D when done. In my case I wanted to test sending syslog data, so I did nc -u <hostname> 514, then wrote test messages. the -u specifies UDP and 514 is the syslog port. I was then able to confirm on the other end the message was received. Handy.