Category Archives: Networking

Fix no internet in KVM/QEMU VMs after installing docker

I ran into a frustrating issue where my KVM VMs would lose network connectivity if I installed docker on my Arch Linux system. After some digging I finally discovered the cause (thanks to anteru.net)

It turns out, docker adds a bunch of iptables rules by default which prevent communication. These will interfere with an already existing bridge, and suddenly your VMs will report no network.

There are two ways to fix this. I went with the route of telling docker to NOT mess with iptables on startup. Less secure, but my system is not directly connected to the internet. I created /etc/docker/daemon.json and added this to it:

{
    "iptables" : false
}

Then restarted my machine. This did the trick!

Restart wireguard interface in OpenWRT

One annoying issue with wireguard in OpenWRT is the fact that it won’t re-check DNS on connection failure. In the event that my public IP changes (dynamic IP) the OpenWRT wireguard client doesn’t ever get the memo, even when DNS is updated.

I discovered here that you can tell OpenWRT via the command line to stop and start the wireguard interface. This forces a new DNS check and then the tunnel builds successfully. The command:

ubus call network.interface.wg0 down &&  ubus call network.interface.wg0 up

Success! Throw this into a cron job and you have an automated failsafe to ensure a reconnect after IP change.

Update 2024-01-16

Here is an example of a cron job to accomplish this:

https://forum.openwrt.org/t/restart-wireguard-via-cli/51935/9

#!/bin/sh
#modified from https://openwrt.org/docs/guide-user/base-system/cron
#modified to use logger for global logging instead of scriptlogfile & added infinite reboot protection for reboot
# Prepare vars
DATE=$(date +%Y-%m-%d" "%H:%M:%S)
#logFile="/persistlogs/syslog"

# Ping and reboot if needed

#YOUR WIREGUARD PEER
CHECKHOSTNAME="192.168.X.X"

notification_email="YOUR@EMAIL.ADRESS"
VPNINTERFACE="wgvpn0"


ping -c3 $CHECKHOSTNAME

if [ $? -eq 0 ]; then
    echo "ok"
    logger $(echo "${DATE} - $0: OK - $VPNINTERFACE UP AND RUNNING")

else
    echo "RESTART wgvpn0 Interface"

Wireguard one-way traffic on USG Pro 4 after dual WAN setup

I have a site-to-site VPN between my Ubiquiti USG Pro-4 and an OpenWRT device over wireguard . It’s worked great until I got a secondary WAN connection as a failover connection since my primary cable connection has been flaky lately.

When you introduce dual-WAN on Ubiquiti devices you have to manually configure everything since the GUI assumes only one WAN connection. I configured my manual DNAT (port forwards) for each interface successfully but struggled to figure out why suddenly my Wireguard VPN between my two sites only went one way (remote side could ping all hosts on local side, but not visa-versa.)

After some troubleshooting I realized the firewall itself could ping the remote subnet just fine, it just wasn’t allowing local hosts to do so. I couldn’t find anything in firewall logs. Eventually I came across this very helpful page from hackad.nu that helped me to solve my problem.

The solution was to add a Firewall Modify rule specifically for the eth0 interface (where all my LAN traffic is routed through) to allow the source address of the subnets I want to traverse the VPN, then apply that modifier to the LAN_IN firewall rule for that interface. I had to do it for any VLANs I wanted to be able to use the Wireguard tunnel as well (vifs of eth0, VLAN 50 in my case)

Here is the relevant config.gateway.json sections, namely “firewall” and “interfaces”:

{
    "firewall": {
        "modify": {
            "Wireguard": {
                "rule": {
                    "10": {
                        "action": "modify",
                        "description": "Allow Wireguard traffic",
                        "modify": {
                            "table": "10"
                        },
                        "source": {
                            "address": "10.1.0.0/16"
                        }
                    }
                }
            }
        },
        "interfaces": {
            "ethernet": {
                "eth0": {
                    "firewall": {
                        "in": {
                            "ipv6-name": "LANv6_IN",
                            "modify": "Wireguard",
                            "name": "LAN_IN"
                        }
                    },
                    "vif": {
                        "50": {
                            "firewall": {
                                "in": {
                                    "ipv6-name": "LANv6_IN",
                                    "modify": "Wireguard",
                                    "name": "LAN_IN"
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

This did the trick! Wireguard is working both directions again, this time with my dual WAN connections.

Prioritize wifi with Network Manager in Arch

My cable internet has been horrid lately. I wanted to be able to hotspot to my phone while maintaining LAN connections to my servers while the cable company takes its sweet time to fix things. Even though I connected to wifi on my phone, my desktop still prioritized the broken connection and wouldn’t use my phone to get to the internet. I verified this by looking at the routing table and running traceroute

sudo ip route
...
default via 10.137.1.1 dev br0 proto dhcp src 10.10.1.124 metric 425 
default via 172.10.10.1 dev wlp69s0 proto dhcp src 172.10.10.4 metric 600 
...

traceroute google.com --max-hops=1
 1  _gateway (10.10.50.1)  0.409 ms  0.449 ms  0.483 ms

The LAN connection’s default gateway had a lower metric than the mobile hotspot connection (lower takes precedence.) To fix this I ran this networkmanager command (thanks to this post for the inspiration)

sudo nmcli connection modify "Nicholas’s iPhone" ipv4.route-metric 50

I noticed DNS traffic was also prioritizing my LAN, which I didn’t want. I fixed it with nmcli as well (thanks to this post)

sudo nmcli connection modify "Nicholas’s iPhone" ipv4.dns-priority 1

I then noticed I couldn’t get to certain LAN subnets. I then realized I needed to add some static routes so they don’t try to go over my hotspot connection (which I learned about here)

sudo nmcli connection modify bridge-br0 +ipv4.routes "10.10.50.0/24 10.10.1.1"

Note you may need to refresh your connection once you’ve made changes. You can either disconnect and reconnect to force a refresh, or run this command (as outlined here.)

sudo nmcli con up bridge-br0 #or whatever your LAN interface name is

Once I refreshed my settings, I was able to get internet via my phone while maintaining all my local network settings.

OpenWRT Wireguard client

My notes on how to configure an OpenWRT device to be a wireguard client (site to site VPN)

More or less follow the instructions from https://openwrt.org/docs/guide-user/services/vpn/wireguard/client

I commented out the IPv6 stuff as well as the pre-shared key. I also had already defined firewall rules so I skipped that section.

One note: make sure your WG_ADDR has the proper subnet mask (I made the mistake of making it a /32 when it needed to be a /24)


# Configuration parameters
WG_IF="wg0"
WG_SERV="remote_wireguard_server_address"
WG_PORT="remote_wireguard_port"
WG_ADDR="wireguard_subnet/wireguard_subnet_mask (/24 for example)"
#WG_ADDR6="fdf1:e8a1:8d3f:9::2/64"

# Generate keys
#umask go=
#wg genkey | tee wgserver.key | wg pubkey > wgserver.pub
#wg genkey | tee wgclient.key | wg pubkey > wgclient.pub
#wg genpsk > wgclient.psk
 
# Client private key
WG_KEY="$(cat wgclient.key)"
 
# Pre-shared key
#WG_PSK="$(cat wgclient.psk)"
 
# Server public key
WG_PUB="public_key_of_wireguard_server"

# Configure network
uci -q delete network.${WG_IF}
uci set network.${WG_IF}="interface"
uci set network.${WG_IF}.proto="wireguard"
uci set network.${WG_IF}.private_key="${WG_KEY}"
uci add_list network.${WG_IF}.addresses="${WG_ADDR}"
#uci add_list network.${WG_IF}.addresses="${WG_ADDR6}"
 
# Add VPN peers
uci -q delete network.wgserver
uci set network.wgserver="wireguard_${WG_IF}"
uci set network.wgserver.public_key="${WG_PUB}"
#uci set network.wgserver.preshared_key="${WG_PSK}"
uci set network.wgserver.endpoint_host="${WG_SERV}"
uci set network.wgserver.endpoint_port="${WG_PORT}"
uci set network.wgserver.route_allowed_ips="1"
uci set network.wgserver.persistent_keepalive="25"
uci add_list network.wgserver.allowed_ips="10.137.50.0/24"
#uci add_list network.wgserver.allowed_ips="::/0"
uci commit network
/etc/init.d/network restart

Wireguard on a USG Pro 4

I ran into some issues when trying to configure an OpenVPN tunnel between my Ubiquiti USG Pro 4 and a Debian VPS. I was very disappointed to discover that the version of OpenVPN on the USG only supports TLS 1.0! My issue was the Debian side rejecting that as insecure.

Thankfully, it was fairly painless to get Wireguard configured on the USG Pro 4. I was hesitant to do so at first because I knew every time my USG was updated I would lose the wireguard pacakge. Fortunately that can be resolved by configuring a post-install script. Thanks to ilar.in and calypte.cc and this github gist for the steps on how to do so.

curl -O https://raw.githubusercontent.com/britannic/install-edgeos-packages/master/install-pkgs
sudo install -o root -g root -m 0755 install-pkgs /config/scripts/post-config.d/install-pkgs
  • Add wireguard DEB package to persistent storage that the script will look for:
sudo mkdir -p /config/data/install-packages
cd /config/data/install-packages
curl -fLSs https://github.com/WireGuard/wireguard-vyatta-ubnt/releases/download/1.0.20200729-1/ugw4-v1-v1.0.20200729-v1.0.20200513.deb
  • Generate a public/private keypair for USG use
cd /config/auth
umask 077
wg genkey > wg_private.key
wg pubkey < wg_private.key > wg_public.key
  • Generate config.gateway.json config to use wireguard
    "interfaces": {
 ...
        "wireguard": {
            "wg0": {
                "address": "<IP_OF_USG_ON_WG_CLIENT_SUBNET>",
                "listen-port": "<WG_LISTEN_PORT>",
                "peer": {
                    "<ENDPOINT_CLIENT_PUBLIC_KEY>": {
                        "allowed-ips": "0.0.0.0/0",
                        "endpoint":  "<ENDPOINT_CLIENT_ADDRESS>:<ENDPOINT_CLIENT_PORT>
"
                    }
                },
                "private-key": "/config/auth/priv.key",
                "route-allowed-ips": false
            },
            "wg1": {
                "address": "<IP_OF_USG_ON_WG_SERVER_SUBNET",
                "firewall": {
                    "in": {
                        "name": "LAN_IN"
                    },
                    "local": {
                        "name": "LAN_LOCAL"
                    },
                    "out": {
                        "name": "LAN_OUT"
                    }
                },
                "listen-port": "<USG_WG_SERVER_LISTEN_PORT>",
                "mtu": "1352",
                "peer": {
                    "<PUBLIC_KEY_OF_WG_CONNECTING_CLIENT": {
                        "allowed-ips": "<SUBNETS_ON_REMOTE_HOST>"
                    }
                },
                "private-key": "/config/auth/wg-server.priv",
                "route-allowed-ips": true
            }
        }
    },

I have two different wireguard interfaces configured – wg0 to be a client to another server, and wg1 to be a server accepting other clients (site-to-site VPN.)

If you want to have multiple peers defined on a single wireguard interface, encapsulate the peers with brackets like so:

"peer": [{
						"--pubkey--": {
							"allowed-ips": [
								"172.255.252.2/32"
							],
							"persistent-keepalive": 60
						}
					},
					{
						"--pubkey--": {
							"allowed-ips": [
								"172.255.252.3/32"
							],
							"persistent-keepalive": 60
						}
					}
				],

Test configuration first

Before committing your config.gateway.json code, test it line by line by SSHing into the USG-Pro 4 and entering config mode. Then type out your JSON lines one at a time, with each key being a new argument separated by a space. The first section above would look like this:

configure
edit interfaces
set wireguard wg0 address WIREGUARD_ADDRESS
set wireguard wg0 listen-port WG_LISTEN_PORT
set wireguard wg0 peer ENDPOINT_CLIENT_PUBLIC_KEY allowed-ips 0.0.0.0/0
set wireguard wg0 peer ENDPOINT_CLIENT_PUBLIC_KEY endpoint ENDPOINT_ADDRESS:ENDPOINT_PORT
set wireguard wg0 private-key /config/auth/priv.key
set wireguard wg0 route-allowed-ips false
commit

If the commit works without error, you can then drop out of the configure section and look at your wireguard config:

exit
sudo wg show

If all looks well, then copy your config.gateway.json to your controller and trigger a reprovision.

Verify after provisioning: 

sudo netstat -npl | grep <WIREGUARD_PORT> | grep udp

Troubleshooting

USG not connecting to changed endpoint address

If you change the address of the wireguard endpoint, USG pro will not connect to that new address. You have to delete and re-create the interface (thanks to https://github.com/Lochnair/vyatta-wireguard/issues/72#issuecomment-423840448 for the information)

Fix this by deleting the wireguard interface

admin@Firewall:~$ configure
[edit]
admin@Firewall# delete interfaces wireguard
[edit]
admin@Firewall# commit


Then reprovision by making a small change, force provision, then change back, and force another provision (annoying) or alternatively reboot firewall.

Wireguard shows established but ping doesn’t work

Example error:

From 10.99.13.1 icmp_seq=5 Destination Host Unreachable
ping: sendmsg: Required key not available

To figure out what’s going on, enable logging to kernel buffer (dmesg) Thanks to procustodibus.com for the info.

echo module wireguard +p > /sys/kernel/debug/dynamic_debug/control

With debug on, ping again and check kernel messages (dmesg)

[Tue Dec 21 22:16:12 2021] wireguard: wg0: No peer has allowed IPs matching 10.99.13.2

This showed I didn’t have my access control properly configured. Modify /etc/wireguard/wg0.conf on your client config and make sure your AllowedIPs are properly letting traffic through.

AllowedIPs = 10.99.13.0/24

USG not allowing connections

Clients unable to connect to USG despite having a good config. Double check your firewall rules. I had neglected to create a WAN LOCAL rule allowing UDP packets on my wireguard port. Once that was configured, handshakes completed successfully.

Firewalls can’t ping each other

I had an issue where the firewalls would pass traffic through, but they couldn’t ping each other. The solution was to add the VPN subnet you created to allowed-ips on both sides of the connection.

Site to site Wireguard VPN between OPNSense & Debian Linux server

I have a Debian linode box acting as a wireguard server. I wanted to join my opnsense firewall to it to allow devices behind it to access the box through the wireguard tunnel. It was not as straightforward as I had hoped, but thankfully I got it all working.

OPNSense side

Documentation link

Install wireguard via GUI

Install the os-wireguard package. Manually drop to the CLI and install the wireguard package as well:
sudo pkg install wireguard

Configure Local instance

  • Name and listen port can be random. Tunnel address is the subnet you wish to expose to the other end (the subnet you wish to have access to the tunnel.)
  • Leave everything else blank and hit save
  • Edit your new connection and copy the Public key, this will need to be sent to the Debian server

Configure Endpoint

  • Name: hostname of Debian server
  • Public Key: Public key of server (can be obtained by running wg show on the server)
  • Shared Secret: blank (unless you’ve configured it on the server)
  • Allowed IPs: IPs or subnets on the Debian server you wish to expose to the client side (the OPNSense box)
  • Endpoint address: DNS name of Debian server
  • Endpoint port: Port Debian wireguard instance is listening on

Enable the VPN

General tab / Enable wireguard checkbox and hit apply.

Debian side

Take down the tunnel

sudo wg-quick down wg0

Edit wireguard config to add peer

sudo vim /etc/wireguard/wg0.conf

[Peer]
PublicKey = <PUBLIC_KEY_YOU_COPIED_IN_LOCAL_INSTANCE_STEP>
AllowedIPs = <IPs or Subnets behind the OPNSense side you wish to be exposed to the Debian side> 

Restart wireguard

sudo wg-quick up wg0

Check connections

Example wg show output below with dummy IPs:

sudo wg show
interface: wg0
  public key: f+/J4JO0aL6kwOaudAvZVa1H2mDzR8Nh3Vfeqq+anF8=
  private key: (hidden)
  listening port: 12345

peer: TuUW7diXcWlaV97z3cQ1/92Zal2Pm9Qz/W2OMN+v20g=
  endpoint: 10.0.0.1:54137
  allowed ips: 10.0.0.2/32
  latest handshake: 17 seconds ago
  transfer: 5.14 KiB received, 3.81 KiB sent

peer: CZuC/+wxvzj9+TiGeyZtcT/lMGZnXsfSs/h5Jtw2VSE=
  endpoint: 8.8.8.8:12345
  allowed ips: 192.168.1.1/32
  latest handshake: 7 minutes, 8 seconds ago
  transfer: 5.89 MiB received, 952.20 MiB sent

The endpoint: line gets populated when a successful VPN connection is made. If it’s missing, the tunnel was not established.

Troubleshooting

OPNSense box

Nothing happens after saving information and enabling tunnel

Make sure latest wireguard package is installed

sudo pkg install wireguard

Get more log output by opening a shell on your OPNSense box and running

sudo /usr/local/etc/rc.d/wireguard start

In my case I was getting this interesting message

[!] Missing WireGuard kernel support (ifconfig: SIOCIFCREATE2: Invalid argument). Falling back to slow userspace implementation.
[#] wireguard-go wg0
┌──────────────────────────────────────────────────────┐
│                                                                                                                                                   │
│   Running wireguard-go is not required because this                                                     │
│   kernel has first class support for WireGuard. For                                                          │
│   information on installing the kernel module,                                                                 │
│   please visit                                                                                                                             │
│         https://www.wireguard.com/install/                                                                           │
└──────────────────────────────────────────────────────┘

I fixed this problem by manually installing wireguard with the pkg install command above.

Debian box

Wireguard config not saving

make sure to stop the tunnel first, otherwise your changes get overwritten by the daemon.

sudo wg-quick down wg0
<make changes>
sudo wg-quick up wg0

piKVM pushover startup script

I’ve had an issue where I wasn’t sure if my dynamic DNS provider registered properly. I then realized that I have a piKVM attached to one of my servers that boots on powerup, even if the server does not. I could utilize this piKVM to help me out.

Thanks to inspiration from Chris Dzombak I was able to whip up a little script that runs on startup. This script waits 5 minutes to allow for my firewall and modem to boot up, then sends a pushover notification to let me know the piKVM is online and what its external IP address is.

To get it working on the piKVM I had to enter into RW mode, write and save the script, add execute permissions to the script, then configure a systemd service to run the script at startup.

Here is the script, saved under /root/boot-pushover.sh

#!/usr/bin/env bash
set -eu

#Wait 5 minutes to allow router bootup
sleep 300

TOKEN="PUSHOVER_APPLICATION_TOKEN"
USER="PUSHOVER_USER_TOKEN"
EXTERNAL_IP="$(curl ifconfig.me)"
MESSAGE="$(hostname) is online. External IP: $EXTERNAL_IP"

#Send pushover command to alert it's up and send its external IP
curl -s \
  --form-string "token=$TOKEN" \
  --form-string "user=$USER" \
  --form-string "message=$MESSAGE" \
  https://api.pushover.net/1/messages.json

Set executable: chmod +x /root/boot-pushover.sh

Here is the systemd service, saved under /etc/systemd/system/boot-pushover-notification.service

[Service]
Type=oneshot
ExecStart=/root/boot-pushover.sh
RemainAfterExit=yes
User=root
Group=root
RestartSec=15
Restart=on-failure

[Unit]
Wants=network.target
After=network.target nss-lookup.target

[Install]
WantedBy=multi-user.target

Reload daemons & enable startup:

systemctl daemon-reload
systemctl enable boot-pushover-notification.service

Test by exiting rw mode and rebooting the piKVM:

ro
reboot

It works really well!

Fix no network after Proxmox 7 upgrade

I upgraded my proxmox server to version 7 and was dismayed to find it had no network connections after a reboot. After much digging I was finally able to find this post which mentioned:

After installing ifupdown2 everything works fine.

Sure enough, ifupdown2 was not installed anymore, and I had configured my networks with it. I had to manually assign an IP address to my node long enough to issue the command
apt install ifupdown2

Once I rebooted, everything came up like it should. Lesson learned: if you use ifupdown2, you must make sure it’s there before you reboot your server!