Tag Archives: OPNsense

Site to site Wireguard VPN between OPNSense & Debian Linux server

I have a Debian linode box acting as a wireguard server. I wanted to join my opnsense firewall to it to allow devices behind it to access the box through the wireguard tunnel. It was not as straightforward as I had hoped, but thankfully I got it all working.

OPNSense side

Documentation link

Install wireguard via GUI

Install the os-wireguard package. Manually drop to the CLI and install the wireguard package as well:
sudo pkg install wireguard

Configure Local instance

  • Name and listen port can be random. Tunnel address is the subnet you wish to expose to the other end (the subnet you wish to have access to the tunnel.)
  • Leave everything else blank and hit save
  • Edit your new connection and copy the Public key, this will need to be sent to the Debian server

Configure Endpoint

  • Name: hostname of Debian server
  • Public Key: Public key of server (can be obtained by running wg show on the server)
  • Shared Secret: blank (unless you’ve configured it on the server)
  • Allowed IPs: IPs or subnets on the Debian server you wish to expose to the client side (the OPNSense box)
  • Endpoint address: DNS name of Debian server
  • Endpoint port: Port Debian wireguard instance is listening on

Enable the VPN

General tab / Enable wireguard checkbox and hit apply.

Debian side

Take down the tunnel

sudo wg-quick down wg0

Edit wireguard config to add peer

sudo vim /etc/wireguard/wg0.conf

[Peer]
PublicKey = <PUBLIC_KEY_YOU_COPIED_IN_LOCAL_INSTANCE_STEP>
AllowedIPs = <IPs or Subnets behind the OPNSense side you wish to be exposed to the Debian side> 

Restart wireguard

sudo wg-quick up wg0

Check connections

Example wg show output below with dummy IPs:

sudo wg show
interface: wg0
  public key: f+/J4JO0aL6kwOaudAvZVa1H2mDzR8Nh3Vfeqq+anF8=
  private key: (hidden)
  listening port: 12345

peer: TuUW7diXcWlaV97z3cQ1/92Zal2Pm9Qz/W2OMN+v20g=
  endpoint: 10.0.0.1:54137
  allowed ips: 10.0.0.2/32
  latest handshake: 17 seconds ago
  transfer: 5.14 KiB received, 3.81 KiB sent

peer: CZuC/+wxvzj9+TiGeyZtcT/lMGZnXsfSs/h5Jtw2VSE=
  endpoint: 8.8.8.8:12345
  allowed ips: 192.168.1.1/32
  latest handshake: 7 minutes, 8 seconds ago
  transfer: 5.89 MiB received, 952.20 MiB sent

The endpoint: line gets populated when a successful VPN connection is made. If it’s missing, the tunnel was not established.

Troubleshooting

OPNSense box

Nothing happens after saving information and enabling tunnel

Make sure latest wireguard package is installed

sudo pkg install wireguard

Get more log output by opening a shell on your OPNSense box and running

sudo /usr/local/etc/rc.d/wireguard start

In my case I was getting this interesting message

[!] Missing WireGuard kernel support (ifconfig: SIOCIFCREATE2: Invalid argument). Falling back to slow userspace implementation.
[#] wireguard-go wg0
┌──────────────────────────────────────────────────────┐
│                                                                                                                                                   │
│   Running wireguard-go is not required because this                                                     │
│   kernel has first class support for WireGuard. For                                                          │
│   information on installing the kernel module,                                                                 │
│   please visit                                                                                                                             │
│         https://www.wireguard.com/install/                                                                           │
└──────────────────────────────────────────────────────┘

I fixed this problem by manually installing wireguard with the pkg install command above.

Debian box

Wireguard config not saving

make sure to stop the tunnel first, otherwise your changes get overwritten by the daemon.

sudo wg-quick down wg0
<make changes>
sudo wg-quick up wg0

Reset root password on OPNSense with ZFS root

A lot of the guides for resetting the root password on an OPNSense box assume a UFS root partition. The password recovery steps do not work if you installed OPNSense with a ZFS root partition. If you try to follow the steps you get a lovely error about “unrecognized filesystem”

The process for ZFS (thanks to this article) is instead to run the following commands:

zfs set readonly=off zroot
zfs mount -a

Once that is done, you can proceed with the rest of the steps. Password recovery steps in full:

  1. Press the number 2 immediately on boot to go into single user mode
  2. Press enter when prompted for shell
  3. Make ZFS read/write:
    1. zfs set readonly=off zroot
    2. zfs mount -a
  4. Reset password
    1. opnsense-shell password
  5. Reboot

Migrating from OPNSense to Ubiquiti Unifi Secure Gateway

I love the Ubiquiti Unifi interface. The only thing missing in my environment was the gateway. I had no complaints with my OPNSense firewall, but that missing section on the Unifi controller homepage haunted me, so I took the plunge and got a Unifi Secure Gateway Pro 4.

Basic Configuration

Initial setup

Official documentation is pretty detailed. Before you install your USG you will want to go into your controller and define your current network by going to Settings / Networks / LAN. This is where you specify DHCP scope and settings. I did not do this and struggled to get DHCP running properly as a result. Be sure to also set NTP settings, as these will also be applied to your USG.

To configure your USG for adoption, hop on the 192.168.1.0/24 network and sign into 192.168.1.1 via a web browser. Username and password are both ubnt. On this screen you can specify WAN and LAN settings. Configure your USG to match the network and gateway settings you’ve defined in your controller and hit apply. Now you can go into your controller and adopt the firewall into your environment.

Firewall

Basic port forwarding rules, static routes, and firewall rules can all be handled in the controller GUI via settings / Routing & Firewall. The GUI assumes your gateway only has one public IP address going to it. If you have multiple public IPs then you will need to configure them in config.gateway.json (see the Advanced Configuration section below.)

DHCP

As stated in the Initial Setup section, this is handled by the controller. You can specify a DHCP scope in the USG’s limited web interface but any settings there are quickly overwritten by the controller pushing out its configuration.

DHCP reservations are handled in the controller via the clients tab (on the left.) Open the client you want to make a reservation for, click the settings cog (top right), click Network, then click “Use Fixed IP Address” and specify the IP you want that device to use.

You can also specify advanced DHCP settings under Settings / Services / DHCP.

Seeing active DHCP leases requires dropping to the CLI on the USG. SSH into the USG and run:

show dhcp leases

Traffic limiting

You can create User Groups in the Unifi interface which define maximum bandwidth usage. You can then assign that User group to a specific client in the Unifi interface.

NAT

The Unifi GUI only supports Destination NAT (DNAT) and only supports the gateway’s WAN IP. You can configure this via settings / Routing & Firewall / Port Forwarding. For more advanced configuration, see below.

Advanced Configuration

A major downside of the USG is that the Unifi interface, while awesome, is extremely limited when it comes to Firewall functions. Thus, most configuration has to be done in the command line to get it to compete with OPNSense.

The core concept with the Unifi ecosystem is that devices are controlled by the Unifi Network Management controller. Thus, with the USG, any changes made to the firewall itself are overwritten by the controller on next provision.

In order to persist any command line changes you make, you must create a config.gateway.json file as outlined here, then copy it to your controller, which will then push the config to your USG on each provision. You will run into problems if you get this json file wrong (reboot loops) so you want to be very sure everything is correct in that file. I recommend a json validator (or an IDE like VS Code.)

One good shortcut I’ve found when googling how to do things is to simply use “edgerouter” instead of “USG” for the search term. The syntax to configure the edgerouter is identical (they both run EdgeOS.)

The most foolproof way to get a config.gateway.json that works is to run the configure commands manually on your USG, then when everything is how you want it, run this command to generate the running config in json format:

mca-ctrl -t dump-cfg > config.txt

You can then read config.txt and look for the specific settings you configured and save them into your config.gateway.json. The JSON syntax follows the CLI commands, with each part of the command broken into different brackets and quotes. An example config.gateway.json looks like this:

{
  "service": {
    "nat": {
      "rule": {
        "4500": {
          "description": "port_forward_WAN2",
          "destination": {
            "address": "100.64.100.100",
            "port": "22"
          },
          "inbound-interface": "eth3",
          "inside-address": {
            "address": "192.168.1.100"
          },
          "protocol": "tcp",
          "type": "destination"
        }
      }
    }
  },
  "vpn": {
    "ipsec": {
      "site-to-site": {
        "peer": {
          "yyyy.ignorelist.com": {
            "authentication": {
              "id": "xxxx.ignorelist.com"
            },
            "local-address": "xxxx.ignorelist.com"
          }
        }
      }
    }
  }
}

DNS

Use the static-host-mapping parameter to specify static DNS entries. Make sure the fqdn is listed in your config, otherwise they may or may not work. Example snippet:

{
...
  "system": {
    "static-host-mapping": {
      "host-name": {
        "firewall": {
          "alias":[
            "firewall.jeppsonlocal"
          ],
          "inet": [
            "192.168.1.1"
          ]
        }
      }
    }
  }
...
}

Live traffic graphs

Sadly there is no live / realtime graphs in the UniFi interface. It’s still possible to get that information if you drop to CLI; however the utilities to see this are not installed by default – you will need to install them (iftop & bmon in my case.) Thanks to this helpful reddit post that got me going.

As of this writing the USG PRO 4 is based in Debian Wheezy, so you will need to add those repositories to the device in order to use apt-get to install iftop & bmon.

Be sure not to get the wrong Debian version. Also be sure not to issue apt-get upgrade – bad things will happen in both cases and you will need to hard reset your device to fix them.

You can add the repositories using the firewall configure command. These can be translated into a config.gateway.json if desired, but I decided not to since this is a pretty low level change that you might not want to happen on future devices. Also note that you will have to re-install these tools after a firmware upgrade.

configure
#Main wheezy archive
set system package repository wheezy components 'main contrib non-free'
set system package repository wheezy distribution wheezy
set system package repository wheezy url 'http://archive.debian.org/debian/'
commit
save
exit

sudo apt-get update
sudo apt-get install iftop bmon

If you want to undo the above changes, substitute set with delete:

#to remove:
configure
delete system package repository wheezy
commit

1:1 NAT

For 1:1 NAT you need 3 NAT rules (Destination NAT, Source NAT, and Hairpin NAT) and a corresponding firewall rule. Example:

{
    "service": {
        "nat": {
            "rule": {
                "1000": {
                    "description": "Mail 1:1 DNAT",
                    "destination": {
                        "address": "1.1.1.1",
                        "port": "25,80,443,465,587,993,995"
                    },
                    "inbound-interface": "pppoe0",
                    "inside-address": {
                        "address": "192.168.1.1"
                    },
                    "protocol": "tcp",
                    "type": "destination"
                },
                "3000": {
                    "description": "Mail 1:1 Hairpin NAT",
                    "destination": {
                        "address": "1.1.1.25",
                        "port": "25,80,443,465,587,993,995"
                    },
                    "inbound-interface": "eth0",
                    "inside-address": {
                        "address": "192.168.1.25"
                    },
                    "protocol": "tcp",
                    "type": "destination"
                },
                "5000": {
                    "description": "Mail 1:1 SNAT",
                    "type": "source",
                    "source": {
                        "address": "192.168.1.25"
                    }
                }
            }
        },
        "firewall": {
            "name": {
                "WAN_IN": {
                    "rule": {
                        "1000": {
                            "action": "accept",
                            "description": "Mail 1:1 DNAT",
                            "destination": {
                                "address": "192.168.1.25",
                                "port": "25,80,443,465,587,993,995"
                            },
                            "protocol": "tcp",
                            "log": "enable"
                        }
                    }
                }
            }
        }
    }
}

OpenVPN Site to Site

My OPNSense router had a site-to-site OpenVPN going with an OpenWRT router. Details on how to configure this are in a separate blog post here.


That covers the basics of what my OPNSense firewall was doing. It’s a bit of a learning curve but once I got past that it’s been working really well.

Site to Site VPN between OPNsense & OpenWRT with Tinc

I’m a real glutton for punishment. I decided to upgrade my parents’ router to OpenWRT. The upgrade went smoothly except for one thing: The VPN I had established between my firewall and theirs.

This was a big enough headache that I even ended up switching my firewall from pfSense to OPNsense (Something I had been contemplating doing for a while anyway) hoping it would make things easier. It didn’t. In the end I abandoned OpenVPN entirely and instead went with Tinc.

Tinc is cool because it’s full mesh peer-to-peer instead of the traditional client / server model. If your equipment supports it, I’d definitely choose it over OpenVPN, especially if multiple sites are involved. A basic rundown of its configuration can be found here.

I used this site as a reference for how to set up tinc.  Essentially you decide on a network name, create private & public keys for each host, and configure each host to connect to each other via a config file & folder structure.

Tinc general configuration

On each device create an /etc/tinc/<network name>/hosts directory structure

mkdir -p /etc/tinc/<network name>/hosts
tincd -n <network name> -K 4096

To configure TINC we need some additional configuration files inside the /etc/tinc/<network name> directory

  • tinc.conf
  • tinc-up (script for bringing up the interface)
  • hosts/<hostname> (one for each location)

tinc.conf can be as simple as this:

Name = <name of host>
ConnectTo = <name of other host> 
#Add each host with an additional ConnectTo line

There needs to be a corresponding file in the hosts directory for each host. Example host file:

Address = <External IP of host>
Subnet = <Subnet other host will share>
#Add more subnets with additional Subnet lines

The host file also need’s the host’s public key. Append it to the end of the file:

cat /etc/tinc/<network name>/rsa_key.pub >> /etc/tinc/<network name>/hosts/<host name>

It’s easiest to generate the host files on each respective host, then copy them to all the other hosts.

The last step is to create the tinc-up script

#!/bin/sh
ubus -t 15 wait_for network.interface.$INTERFACE
ip link set $INTERFACE up ip addr add 172.16.0.1/24 dev $INTERFACE

Modify the IP used on each host so they don’t overlap. The private network here is what’s used for inter-host communication.

Make the script executable:

chmod 755 /etc/tinc/<network name>/tinc-up

OPNSense specific configuration

I got this working through an enhanced tinc package for OPNsense located here.  I will copypasta the content from that site here for easier reference:

Installation

The version might change, adjust it if fetch fails

fetch https://raw.githubusercontent.com/EugenMayer/tinc-opnsense/master/dist/os-tincdcustom-latest.txz
pkg install os-tincdcustom-latest.txz

1. your network

  1. copy the /usr/local/etc/tinc/example folder to /usr/local/etc/tinc/yournetwork
  2. enter yournetwork into /usr/local/etc/tinc/nets.boot to let this network be started on boot
  3. create keypairs by runng tincd -n <yournetwork> -K

2. your network configuration and tun device

  1. Edit /usr/local/etc/tinc/yournetwork/tinc.conf set the server you want to connect to and how this server is to be named
  2. Edit /usr/local/etc/tinc/yournetwork/tinc-up and adjust the network/netbitmask

3. finally the host configuration

  1. enter the /usr/local/etc/tinc/yournetwork/hosts folder and rename the files according to what you have chosen for youservername and theotherservername – they must match!
  2. enter the public key of the “this server” you find under /usr/local/etc/tinc/yournetwork/ into the according thisservernamefile and adjust the subnet this server offers (or subnets)
  3. enter the public key of the “other server” into the according theotherservername file and adjust the subnet the other server offers (or subnets)

4. OPNsense Interface/Gateway/Route/FW configuration

Please see this answer for a brief description

  • You need to create a Gateway, which is configured to go through tinc0 with “dynamic” (do not enter an IP on Gateway field)
  • You need to add a route to <remote subnet> through this gateway
  • Add your tinc0 interface in the Interface section. You can configure a ipv4 address or you don’t, does not matter. If you do, use your tinc-up configured address. Doing this enabled you to create FW Rules for the Tinc interface – which we will need.
  • Add a firewall on the Tinc interface to allow communication to local & remote subnets
    • Alternatively, add a single rule for the Tinc interface to allow any/any access (lazy, less secure)
  • Don’t forget to create a firewall rule allowing the port you’ve configured tinc to run on access from the internet.

OpenWRT specific configuration

Openwrt follows the general tinc configuration exactly. Make the appropriate folders and config files in /etc/tinc/<network name>/ and then test your configuration:

tincd -n <network name>

Once connection is established and working:

Create interface for your VPN (network / interfaces / add new interface) Select the name of your tinc network name from the list.

Next bridge your VPN to the LAN by going to Network / Firewall and editing your LAN zone. Select your VPN interface created earlier from the list and hit save & apply.

Run on startup

I could not find clear documentation on getting this to work on startup. There is a startup script for tinc but it doesn’t appear to launch my tinc config. I ended up modifying /etc/init.d/tinc and adding these lines to the start() and stop() functions. You could also just write your own simple init script to accomplish this.

start() {
...
/usr/sbin/tincd -n <network name>
}

stop() {
...
kill `pidof tincd`
}

Troubleshooting

Tincdcustom service won’t start in OPNSense

Starting from the GUI just does nothing, starting from CLI reveals this unhelpful error:

configctl tincdcustom status

Error (1)

From the OPNSense docs I determined which command I can run to see exactly why. The command is located in this configd configuration file: /usr/local/opnsense/service/conf/actions.d/actions_tincdcustom.conf

command:/usr/local/etc/rc.d/tincdcustom start

Doing that command manually revealed what the problem was:

/usr/local/etc/rc.d/tincdcustom start

Please create /usr/local/etc/tinc/nets.boot.

I had skipped step 1.2 of the tincdcustom instalaltion guide:

enter yournetwork into /usr/local/etc/tinc/nets.boot to let this network be started on boot

Once I added a single word – the name of the network I want to start on bootup – to /usr/local/etc/tinc/nets.boot – the daemon started and worked properly.

Running Tinc in verbose mode

Coming from the tinc documentation, I ran tinc in verbose mode on both of my hosts to troubleshoot why a connection wasn’t happening. It was very helpful.

tincd -n netname -d5 -D

Edit 9/9/2018

I had issues with my tinc startup script not working on the openwrt side. I found here that stated you should add this line to the top of your tinc-up config:

ubus -t 15 wait_for network.interface.$INTERFACE

This solved the startup issue for me.