Category Archives: Networking

Networking, OS

NPM reboot workaround script

Leave a comment

I had an annoying “chicken and egg” problem with nginx proxy manager. It has hosts configurations that reference DNS names that are only reachable over VPN. It also hosts headscale, which is required for VPN to establish properly for said DNS entries. Brought up from scratch (say, after a reboot) NPM fails to launch because it can’t resolve the DNS entries. Headscale therefore isn’t reachable by the other nodes, and we’re stuck.

I developed a script to be run at boot to get around this annoying behavior of nginx proxy manager. It disables all other hosts in NPM except headscale, which doesn’t need DNS resolution as it resides on the same host. It then fires up NPM, and then tailscale, and waits to confirm the VPN is established and DNS resolution is working. Then it restores all the other hosts and restarts NPM. This script was 100% done myself, no AI assistance was used.

#!/bin/bash
# Script to properly bring things up after a reboot
# Name resolution doesn't happen until vpn VIP comes up
# NPM doesn't spin up until name resolution works
# vpn headscale is part of NPM
#
# Replace all VPN DNS host entries with 127.0.0.1
# Wait for NPM to come up
# Restore original host entries to get name resolution to work
# restart npm
#
# Run on a cron job @reboot. Ensure cron service is enabled.
# @reboot /docker/npm/reboot-script.sh | tee /tmp/reboot-script.log

# Variables
DOCKER_DIR=/docker/npm
CONF_DIR=$DOCKER_DIR/data/nginx/proxy_host
TEMP_DIR=/tmp/$(basename $CONF_DIR)
VPN_IP=100.1.1.1

# Log start of script
echo "Reboot script started on $(date)"

# Backup host entries
rsync -aP --delete $CONF_DIR $(dirname $TEMP_DIR) 

# Disable all hosts except for headscale
sed -i  's/\".*.<VPN_DNS_SUFFIX>\"/\"127.0.0.1\"/g' $CONF_DIR/*.conf
sed -i  's/http:\/\/.*.<VPN_DNS_SUFFIX>/http:\/\/127.0.0.1/g' $CONF_DIR/*.conf

# Listen on 127.0.0.1 for npm
sed -i "s/$VPN_IP/127.0.0.1/g" $DOCKER_DIR/docker-compose.yml

# Stop and restart npm
cd $CONF_DIR && docker compose down && docker compose up -d

# Wait for NPM to launch
until netstat -an|grep 0.0.0.0:80 >/dev/null; do echo "Waiting for NPM"; sleep 5; done; echo "NPM up"

# Restart tailscale
systemctl restart tailscaled

# Wait until VPN pings are successful
until ping -w 1 -c 1 <VPN_DNS_IP> >/dev/null; do echo "Waiting for VPN"; sleep 5; done; echo "VPN Successful"

# Restore config files
rsync -aP $TEMP_DIR/ $CONF_DIR
sed -i "s/127.0.0.1/$VPN_IP/g" $DOCKER_DIR/docker-compose.yml

# Restart npm
cd $CONF_DIR && docker compose down && docker compose up -d

echo "Reboot script completed on $(date)"

Networking

Site to Site VPN with Tailscale subnet router

Leave a comment

My manual wireguard site to site solution worked but had latency issues. I wanted a more streamlined way to get my site to site VPN working properly. I decided to finally try out tailscale but didn’t want to rely on their servers, so I spun up headscale and hosted the control server myself.

My sites have disparate routers which don’t lend to installing the tailscale client, so I opted to spin up dedicated subnet router nodes and then tell the firewalls at each site to forward the routes for the other sites’ subnets to their local subnet router.

The documentation is quite good and it didn’t take long for me to get a working solution.

Configuration

  • Install headscale
    • Configure URL, DNS, ACL
    • Allow all: {}
    • sudo docker exec headscale <command>
  • Set up subnet routers
    • Advertise routes: sudo tailscale set --advertise-routes=192.0.2.0/24,198.51.100.0/24
  • Advertise exit node, specify login server, set hostname, accept routes
    • sudo tailscale up --hostname <HOSTNAME> --login-server=<HEADSCALE_URL> --accept-routes --advertise-exit-node
  • Accept routing on the control server
    • sudo docker exec headscale headscale nodes list-route
    • sudo docker exec headscale headscale nodes approve-routes -i <ID> -r <SUBNET>
  • Add tailscale interface as trusted interface
    • sudo firewall-cmd --zone=trusted --add-interface=tailscale0 --permanent
  • Configure docker DNS to use tailscale’s magic DNS
    • /etc/docker/daemon.json:
      {"dns": ["100.100.100.100","1.1.1.1","9.9.9.9"]}

Troubleshooting

CONFIG_TUN error

is CONFIG_TUN enabled in your kernel? modprobe tun failed with: modprobe: FATAL: Module tun not found in directory /lib/modules/6.8.8-4-pve

Solution found here: https://diegocarrasco.com/install-tailscale-proxmox-lxc-container-almalinux-9

You need to edit the conf file for your LXC and allow/mount /dev/net/tun to your container:

vi /etc/pve/lxc/<LXC_NUMBER>.conf
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

CLI, Networking

Port Forward from Internet to Wireguard interface

1 Comment

I needed to give my CGNAT-backed home internet a way to have a public IP address. My first solution was to use wireguard directly, and forward ports as needed. I came across this article that helped me do it. The key was to enable packed masquerading so the return path could be completed. Example wireguard server config:

# packet forwarding
PreUp = sysctl -w net.ipv4.ip_forward=1

# port forwarding
PreUp = iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 2000 -j DNAT --to-destination 10.0.0.1:8080
PostDown = iptables -t nat -D PREROUTING -i eth0 -p tcp --dport 2000 -j DNAT --to-destination 10.0.0.1:8080

# packet masquerading
PreUp = iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
PostDown = iptables -t nat -D POSTROUTING -o wg0 -j MASQUERADE

Example wireguard client config:

PreUp = iptables -t nat -A POSTROUTING -o wg0
PostUp = iptables -A FORWARD -i %i -j ACCEPT
PostDown = iptables -D FORWARD -i %i -j ACCEPT
PostDown = iptables -t nat -D POSTROUTING -o wg0

Make sure you have correct allowedIPs configured on client and server. This does work, but it shows the source IP as being the VPN destination. If you value seeing what true external source IPs are, then this solution is not for you (eg seeing external IPs accessing a webserver.)

DNS resolution inside docker containers

I had an issue where docker containers weren’t resolving DNS properly over this VPN tunnel. I found this site that explained I needed to update my docker daemon.json to explicitly specify which DNS servers to use, then restart docker:

{
  "dns": ["172.17.0.1","10.10.10.1"]
}
Networking

Prometheus dashboard for MB8611 cable modem statistics

Leave a comment

I’ve had issues with my cable internet service lately. I came across this excellent guide from Duckware to troubleshooting cable internet issues and realized my modem had all the information I needed to troubleshoot my connection. Calling the cable company is a complete waste of time. Fortunately I was able to fix it with some insight into what was going on. In my case, it ended up being a loose cable. I simply had to tighten every connection along the way with a simple tool.

In my troubleshooting I was on the hunt for a Prometheus exporter that would get the data from my cable modem, as manually getting it from the modem webpage is painful. The prometheus-moto-exporter was fairly straightforward to set up. I just needed a dashboard to actually view and interpret the data. There was reference to this project but it appears to have been deleted. I eventually found that at some point someone had forked it here. It got me mostly there, but there was a syntax error in the JSON, and it referred to the exported data by a different name. I was able to fix both issues and created my own fork, located here: https://github.com/jimmyface/prometheus-mb8611-dashboard (note: it looks like the original repo made a recent update fixing the quotation problem.)

It works! I now have a quick and easy way to interpret data from my cable modem, and it is beautiful.

Networking

Wireguard client full tunnel no internet

Leave a comment

I tried to set up a wireguard full tunnel VPN on my Debian Bookworm server and ran into issues with internet connectivity. LAN / VPN connectivity was fine, just no internet.

My first realization was that when making changes to your config, be sure to wg-quick down & wg-quick up (if you’re using wg-quick.) Simply editing the files and reloading the service doesn’t pick up the changes.

I followed this guide to get it up, and it simply wasn’t working:

https://wiki.debian.org/WireGuard#Step_2_-Alternative_A-_Manual_Configuration

I did eventually realize I need to enable the following in /etc/sysctl.conf

net.ipv4.ip_forward = 1

Then reload settings with:

# sysctl -p

In the wireguard server config I added these iptables commands:
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

I discovered that allowed IPs on the server side are simply the IP address(es) of the wireguard clients, nothing more.

For full tunnel, set Client’s allowed IP to 0.0.0.0/0

I did all this and it still didn’t work. Then I stumbled upon https://www.ckn.io/blog/2017/11/14/wireguard-vpn-typical-setup

which mentioned to enable conntrack with iptables:

iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

When I enabled conntrack, internet connectivity worked. I decided to reboot without making the above iptables commands persistent. But it worked after reboot!

Lesson learned: try rebooting the host as a wireguard troubleshooting step, especially if all the configs look like they should be working but simply aren’t.

Here are my working configs:

Server:

[Interface]
Address = 10.10.1.1/24
SaveConfig = true
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
ListenPort = 51820
PrivateKey = <server_private_key>

[Peer]
PublicKey = <client_public_key>
AllowedIPs = 10.10.1.5/32

Client:

[Interface]
Address = 10.10.1.5/32
PrivateKey = <private key>
DNS = 10.10.1.1 10.10.1.2

[Peer]
PublicKey = <server public key>
Endpoint = mx.jeppson.org:54137
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 21

Networking, Virtualization

Fix no internet in KVM/QEMU VMs after installing docker

2 Comments

I ran into a frustrating issue where my KVM VMs would lose network connectivity if I installed docker on my Arch Linux system. After some digging I finally discovered the cause (thanks to anteru.net)

It turns out, docker adds a bunch of iptables rules by default which prevent communication. These will interfere with an already existing bridge, and suddenly your VMs will report no network.

There are two ways to fix this. I went with the route of telling docker to NOT mess with iptables on startup. Less secure, but my system is not directly connected to the internet. I created /etc/docker/daemon.json and added this to it:

{
    "iptables" : false
}

Then restarted my machine. This did the trick!

CLI, Networking

Restart wireguard interface in OpenWRT

3 Comments

One annoying issue with wireguard in OpenWRT is the fact that it won’t re-check DNS on connection failure. In the event that my public IP changes (dynamic IP) the OpenWRT wireguard client doesn’t ever get the memo, even when DNS is updated.

I discovered here that you can tell OpenWRT via the command line to stop and start the wireguard interface. This forces a new DNS check and then the tunnel builds successfully. The command:

ubus call network.interface.wg0 down &&  ubus call network.interface.wg0 up

Success! Throw this into a cron job and you have an automated failsafe to ensure a reconnect after IP change.

Update 2024-01-16

Here is an example of a cron job to accomplish this:

https://forum.openwrt.org/t/restart-wireguard-via-cli/51935/9

#!/bin/sh
#modified from https://openwrt.org/docs/guide-user/base-system/cron
#modified to use logger for global logging instead of scriptlogfile & added infinite reboot protection for reboot
# Prepare vars
DATE=$(date +%Y-%m-%d" "%H:%M:%S)
#logFile="/persistlogs/syslog"

# Ping and reboot if needed

#YOUR WIREGUARD PEER
CHECKHOSTNAME="192.168.X.X"

notification_email="YOUR@EMAIL.ADRESS"
VPNINTERFACE="wgvpn0"


ping -c3 $CHECKHOSTNAME

if [ $? -eq 0 ]; then
    echo "ok"
    logger $(echo "${DATE} - $0: OK - $VPNINTERFACE UP AND RUNNING")

else
    echo "RESTART wgvpn0 Interface"
Networking

Wireguard one-way traffic on USG Pro 4 after dual WAN setup

Leave a comment

I have a site-to-site VPN between my Ubiquiti USG Pro-4 and an OpenWRT device over wireguard . It’s worked great until I got a secondary WAN connection as a failover connection since my primary cable connection has been flaky lately.

When you introduce dual-WAN on Ubiquiti devices you have to manually configure everything since the GUI assumes only one WAN connection. I configured my manual DNAT (port forwards) for each interface successfully but struggled to figure out why suddenly my Wireguard VPN between my two sites only went one way (remote side could ping all hosts on local side, but not visa-versa.)

After some troubleshooting I realized the firewall itself could ping the remote subnet just fine, it just wasn’t allowing local hosts to do so. I couldn’t find anything in firewall logs. Eventually I came across this very helpful page from hackad.nu that helped me to solve my problem.

The solution was to add a Firewall Modify rule specifically for the eth0 interface (where all my LAN traffic is routed through) to allow the source address of the subnets I want to traverse the VPN, then apply that modifier to the LAN_IN firewall rule for that interface. I had to do it for any VLANs I wanted to be able to use the Wireguard tunnel as well (vifs of eth0, VLAN 50 in my case)

Here is the relevant config.gateway.json sections, namely “firewall” and “interfaces”:

{
    "firewall": {
        "modify": {
            "Wireguard": {
                "rule": {
                    "10": {
                        "action": "modify",
                        "description": "Allow Wireguard traffic",
                        "modify": {
                            "table": "10"
                        },
                        "source": {
                            "address": "10.1.0.0/16"
                        }
                    }
                }
            }
        },
        "interfaces": {
            "ethernet": {
                "eth0": {
                    "firewall": {
                        "in": {
                            "ipv6-name": "LANv6_IN",
                            "modify": "Wireguard",
                            "name": "LAN_IN"
                        }
                    },
                    "vif": {
                        "50": {
                            "firewall": {
                                "in": {
                                    "ipv6-name": "LANv6_IN",
                                    "modify": "Wireguard",
                                    "name": "LAN_IN"
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

This did the trick! Wireguard is working both directions again, this time with my dual WAN connections.

CLI, Networking

Prioritize wifi with Network Manager in Arch

Leave a comment

My cable internet has been horrid lately. I wanted to be able to hotspot to my phone while maintaining LAN connections to my servers while the cable company takes its sweet time to fix things. Even though I connected to wifi on my phone, my desktop still prioritized the broken connection and wouldn’t use my phone to get to the internet. I verified this by looking at the routing table and running traceroute

sudo ip route
...
default via 10.137.1.1 dev br0 proto dhcp src 10.10.1.124 metric 425 
default via 172.10.10.1 dev wlp69s0 proto dhcp src 172.10.10.4 metric 600 
...

traceroute google.com --max-hops=1
 1  _gateway (10.10.50.1)  0.409 ms  0.449 ms  0.483 ms

The LAN connection’s default gateway had a lower metric than the mobile hotspot connection (lower takes precedence.) To fix this I ran this networkmanager command (thanks to this post for the inspiration) 

sudo nmcli connection modify "Nicholas’s iPhone" ipv4.route-metric 50

I noticed DNS traffic was also prioritizing my LAN, which I didn’t want. I fixed it with nmcli as well (thanks to this post)

sudo nmcli connection modify "Nicholas’s iPhone" ipv4.dns-priority 1

I then noticed I couldn’t get to certain LAN subnets. I then realized I needed to add some static routes so they don’t try to go over my hotspot connection (which I learned about here)

sudo nmcli connection modify bridge-br0 +ipv4.routes "10.10.50.0/24 10.10.1.1"

Note you may need to refresh your connection once you’ve made changes. You can either disconnect and reconnect to force a refresh, or run this command (as outlined here.) 

sudo nmcli con up bridge-br0 #or whatever your LAN interface name is

Once I refreshed my settings, I was able to get internet via my phone while maintaining all my local network settings.