Hardware, Networking, Virtualization

Add backup link for Proxmox cluster

Leave a comment

I kept having a frustrating issue where my 40gig link for all hosts in my 3-node mesh network would go down if one host became unresponsive. This would cause the healthy nodes to fence themselves and take the whole cluster down until I rebooted them. I needed a way to fall back to my 1gig links if the 40gig failed.

I landed on this writeup which is pretty straightforward. To add secondary links after the cluster has been created, you simply need to modify /etc/pve/corosync.conf. Increment the version number, add the backup addresses per node (ring1_addr), and add the link to the base cluster config (interface link number: 1). The priority numbers are abitrary. The higher the priority number, the more preferred it is.

root@pve-a:/etc/pve# cat corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve-a
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.255.11
    ring1_addr: 192.168.4.111
  }
  node {
    name: pve-b
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.255.12
    ring1_addr: 192.168.4.112
  }
  node {
    name: pve-c
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 192.168.255.13
    ring1_addr: 192.168.4.113
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: pve-cluster-a
  config_version: 4
  interface {
    linknumber: 0
    knet_link_priority: 255
  }
  interface {
    linknumber: 1
    knet_link_priority: 4
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

If you get an error that the file is read only, ensure your cluster has quorum: pvecm status|grep Quorate

The failures that caused this for me are complete system hangs. They wouldn’t happen if the system powered off suddenly – frr handles that fine already. For whatever reason a system lockup caused frr to lose its mind and crash. The cluster remained working but vtysh would simply hang. Once the frozen host was rebooted, the 40g links would fail completely. I would have to restart the frr service to bring things back up. As a stopgap, I added restart=always to the frr systemd service file as per this writeup. 

systemctl edit frr
[Service]
Restart=always
RestartSec=5s

After implementing a backup interface, the next system hang did not take down the cluster!

Networking

Secure port forwarding with ssh & systemd

Leave a comment

I wanted an easy way to forward ports from my VPS to a host on my local network. Firewalld and iptables were giving me grief, so I settled on using ssh port forwarding instead. I came across this gist which was super helpful. Now instead of dealing with firewall and iptables, I simply have an ssh session spun up as a systemd service. If the connection gets dropped or killed it auto connects again. The port forwards come through nice and clean. Brilliant.

I had to configure ssh keys to allow for passwordless connection. Here’s my systemd file (I went the lazy route and didn’t configure environment variables in /etc/system/default like the gist suggests)

[Unit]
Description=Setup a secure tunnel to LAN_HOST
After=network.target

[Service]
Environment="LOCAL_ADDR=<REMOTE_IP_OF_VPS>"
ExecStart=/usr/bin/ssh -i /home/ssh_user/.ssh/id_rsa -NT -o ServerAliveInterval=60 -o ExitOnForwardFailure=yes -L ${LOCAL_ADDR}:VPS_PORT:localhost:LAN_PORT -L ${LOCAL_ADDR}:VPS_PORT_2:localhost:LAN_PORT_2 ssh_user@LAN_HOST

# Restart every >2 seconds to avoid StartLimitInterval failure
RestartSec=5
Restart=always

[Install]
WantedBy=multi-user.target

Put that in /etc/systemd/system and do a systemctl daemon-reload. Ensure your firewall has those ports open (including firewall-reload.) Then enable & start it. Profit.

Networking, OS

NPM reboot workaround script

Leave a comment

I had an annoying “chicken and egg” problem with nginx proxy manager. It has hosts configurations that reference DNS names that are only reachable over VPN. It also hosts headscale, which is required for VPN to establish properly for said DNS entries. Brought up from scratch (say, after a reboot) NPM fails to launch because it can’t resolve the DNS entries. Headscale therefore isn’t reachable by the other nodes, and we’re stuck.

I developed a script to be run at boot to get around this annoying behavior of nginx proxy manager. It disables all other hosts in NPM except headscale, which doesn’t need DNS resolution as it resides on the same host. It then fires up NPM, and then tailscale, and waits to confirm the VPN is established and DNS resolution is working. Then it restores all the other hosts and restarts NPM. This script was 100% done myself, no AI assistance was used.

#!/bin/bash
# Script to properly bring things up after a reboot
# Name resolution doesn't happen until vpn VIP comes up
# NPM doesn't spin up until name resolution works
# vpn headscale is part of NPM
#
# Replace all VPN DNS host entries with 127.0.0.1
# Wait for NPM to come up
# Restore original host entries to get name resolution to work
# restart npm
#
# Run on a cron job @reboot. Ensure cron service is enabled.
# @reboot /docker/npm/reboot-script.sh | tee /tmp/reboot-script.log

# Variables
DOCKER_DIR=/docker/npm
CONF_DIR=$DOCKER_DIR/data/nginx/proxy_host
TEMP_DIR=/tmp/$(basename $CONF_DIR)
VPN_IP=100.1.1.1

# Log start of script
echo "Reboot script started on $(date)"

# Backup host entries
rsync -aP --delete $CONF_DIR $(dirname $TEMP_DIR) 

# Disable all hosts except for headscale
sed -i  's/\".*.<VPN_DNS_SUFFIX>\"/\"127.0.0.1\"/g' $CONF_DIR/*.conf
sed -i  's/http:\/\/.*.<VPN_DNS_SUFFIX>/http:\/\/127.0.0.1/g' $CONF_DIR/*.conf

# Listen on 127.0.0.1 for npm
sed -i "s/$VPN_IP/127.0.0.1/g" $DOCKER_DIR/docker-compose.yml

# Stop and restart npm
cd $CONF_DIR && docker compose down && docker compose up -d

# Wait for NPM to launch
until netstat -an|grep 0.0.0.0:80 >/dev/null; do echo "Waiting for NPM"; sleep 5; done; echo "NPM up"

# Restart tailscale
systemctl restart tailscaled

# Wait until VPN pings are successful
until ping -w 1 -c 1 <VPN_DNS_IP> >/dev/null; do echo "Waiting for VPN"; sleep 5; done; echo "VPN Successful"

# Restore config files
rsync -aP $TEMP_DIR/ $CONF_DIR
sed -i "s/127.0.0.1/$VPN_IP/g" $DOCKER_DIR/docker-compose.yml

# Restart npm
cd $CONF_DIR && docker compose down && docker compose up -d

echo "Reboot script completed on $(date)"

CLI

Get a summary of disk usage from select files with find, sed, du, and xargs

Leave a comment

I wanted a quick way in the command line to get the disk usage of a bunch of zip files I downloaded in the previous day. I also wanted them sorted by filename and to have quotes surround each filename. I learned from this stackexchange post that du -ch is the command I want to accomplish this. Here is my final command. It works! Note: I ran this on a mac, so I had to use gsed because the version of sed that ships with mac is rather crippled. On linux the command would simply be sed instead of gsed

find . -name "*.zip" -mtime -1|sort -h|sed 's/.\//"/g'|sed 's/.zip/.zip"/g'|gsed -z 's/\n/ /g'|xargs du -ch

The output looks like this (snippet – not the full output):

753M V-A – Mixed by Mahiane – OXYCANTA.zip
912M V-A – Mixed by Nova – ALBEDO.zip
816M V-A – Selected by Fishimself – AMBROSIA (24bits).zip
977M Various Artists – FAHRENHEIT PROJECT – Part 1.zip
992M Various Artists – FAHRENHEIT PROJECT – Part 2.zip
848M Various Artists – FAHRENHEIT PROJECT – Part 3.zip
849M Various Artists – FAHRENHEIT PROJECT – Part 4.zip
817M Various Artists – FAHRENHEIT PROJECT – Part 5.zip
897M Various Artists – FAHRENHEIT PROJECT – Part 6.zip
897M Various Artists – FAHRENHEIT PROJECT – Part 7.zip
737M Various Artists – ISOLATED (24bit).zip
817M Various Artists – OPIA (24bit).zip
55G total

For the curious, I had purchased the Ultimae Digital Collection. Great stuff.

Networking

Site to Site VPN with Tailscale subnet router

Leave a comment

My manual wireguard site to site solution worked but had latency issues. I wanted a more streamlined way to get my site to site VPN working properly. I decided to finally try out tailscale but didn’t want to rely on their servers, so I spun up headscale and hosted the control server myself.

My sites have disparate routers which don’t lend to installing the tailscale client, so I opted to spin up dedicated subnet router nodes and then tell the firewalls at each site to forward the routes for the other sites’ subnets to their local subnet router.

The documentation is quite good and it didn’t take long for me to get a working solution.

Configuration

  • Install headscale
    • Configure URL, DNS, ACL
    • Allow all: {}
    • sudo docker exec headscale <command>
  • Set up subnet routers
    • Advertise routes: sudo tailscale set --advertise-routes=192.0.2.0/24,198.51.100.0/24
  • Advertise exit node, specify login server, set hostname, accept routes
    • sudo tailscale up --hostname <HOSTNAME> --login-server=<HEADSCALE_URL> --accept-routes --advertise-exit-node
  • Accept routing on the control server
    • sudo docker exec headscale headscale nodes list-route
    • sudo docker exec headscale headscale nodes approve-routes -i <ID> -r <SUBNET>
  • Add tailscale interface as trusted interface
    • sudo firewall-cmd --zone=trusted --add-interface=tailscale0 --permanent
  • Configure docker DNS to use tailscale’s magic DNS
    • /etc/docker/daemon.json:
      {"dns": ["100.100.100.100","1.1.1.1","9.9.9.9"]}

Troubleshooting

CONFIG_TUN error

is CONFIG_TUN enabled in your kernel? modprobe tun failed with: modprobe: FATAL: Module tun not found in directory /lib/modules/6.8.8-4-pve

Solution found here: https://diegocarrasco.com/install-tailscale-proxmox-lxc-container-almalinux-9

You need to edit the conf file for your LXC and allow/mount /dev/net/tun to your container:

vi /etc/pve/lxc/<LXC_NUMBER>.conf
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

CLI, Networking

Port Forward from Internet to Wireguard interface

1 Comment

I needed to give my CGNAT-backed home internet a way to have a public IP address. My first solution was to use wireguard directly, and forward ports as needed. I came across this article that helped me do it. The key was to enable packed masquerading so the return path could be completed. Example wireguard server config:

# packet forwarding
PreUp = sysctl -w net.ipv4.ip_forward=1

# port forwarding
PreUp = iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 2000 -j DNAT --to-destination 10.0.0.1:8080
PostDown = iptables -t nat -D PREROUTING -i eth0 -p tcp --dport 2000 -j DNAT --to-destination 10.0.0.1:8080

# packet masquerading
PreUp = iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
PostDown = iptables -t nat -D POSTROUTING -o wg0 -j MASQUERADE

Example wireguard client config:

PreUp = iptables -t nat -A POSTROUTING -o wg0
PostUp = iptables -A FORWARD -i %i -j ACCEPT
PostDown = iptables -D FORWARD -i %i -j ACCEPT
PostDown = iptables -t nat -D POSTROUTING -o wg0

Make sure you have correct allowedIPs configured on client and server. This does work, but it shows the source IP as being the VPN destination. If you value seeing what true external source IPs are, then this solution is not for you (eg seeing external IPs accessing a webserver.)

DNS resolution inside docker containers

I had an issue where docker containers weren’t resolving DNS properly over this VPN tunnel. I found this site that explained I needed to update my docker daemon.json to explicitly specify which DNS servers to use, then restart docker:

{
  "dns": ["172.17.0.1","10.10.10.1"]
}
Web

Troubleshoot blackbox exporter errors

Leave a comment

I had a frustrating issue where prometheus blackbox exporter wasn’t able to check one of my websites and I couldn’t figure out why. I finally found this site which explained you can append &debug=true to the end of your probe in a URL string talking directly to blackbox. For example:

http://prometheus:9115/probe?module=http_2xx&target=http://customsite.com:8096/web/&debug=true

That finally got me to see what the problem was. It was resolving to an IPv6 address, but I didn’t have my IPv6 stack properly configured. I then discovered this site which led me to the solution: prefer IPv4. I appended this to my http_2xx config:

http_2xx:
  prober: http
  http:
    preferred_ip_protocol: ip4
...

That fixed the issue!

Networking

Prometheus dashboard for MB8611 cable modem statistics

Leave a comment

I’ve had issues with my cable internet service lately. I came across this excellent guide from Duckware to troubleshooting cable internet issues and realized my modem had all the information I needed to troubleshoot my connection. Calling the cable company is a complete waste of time. Fortunately I was able to fix it with some insight into what was going on. In my case, it ended up being a loose cable. I simply had to tighten every connection along the way with a simple tool.

In my troubleshooting I was on the hunt for a Prometheus exporter that would get the data from my cable modem, as manually getting it from the modem webpage is painful. The prometheus-moto-exporter was fairly straightforward to set up. I just needed a dashboard to actually view and interpret the data. There was reference to this project but it appears to have been deleted. I eventually found that at some point someone had forked it here. It got me mostly there, but there was a syntax error in the JSON, and it referred to the exported data by a different name. I was able to fix both issues and created my own fork, located here: https://github.com/jimmyface/prometheus-mb8611-dashboard (note: it looks like the original repo made a recent update fixing the quotation problem.)

It works! I now have a quick and easy way to interpret data from my cable modem, and it is beautiful.

OS, Virtualization

Fix frozen boot / no network after upgrading to Proxmox 9

Leave a comment

I recently upgraded to Proxmox Virtual Environment 9 and was dismayed to see the server just hung forever after saying disks were initialized. A boot in debug mode (advanced grub boot option) revealed that systemd was hanging forever on waiting for the network service to start.

My first trick was to give the networking service a timeout by modifying /etc/systemd/system/network-online.target.wants/networking.service and adding TimeoutStartSec=90sec to the [Service] section. This at least allowed the system to boot normally and not hang anymore.

After boot, however, networking did not work. After much troubleshooting I found this article explaining it has to do with a post-up command I had for my mesh networking interface: post-up /usr/bin/systemctl restart frr.service

Once I removed that line, I was able to boot normally.