Category Archives: Networking

Migrating from OPNSense to Ubiquiti Unifi Secure Gateway

I love the Ubiquiti Unifi interface. The only thing missing in my environment was the gateway. I had no complaints with my OPNSense firewall, but that missing section on the Unifi controller homepage haunted me, so I took the plunge and got a Unifi Secure Gateway Pro 4.

Basic Configuration

Initial setup

Official documentation is pretty detailed. Before you install your USG you will want to go into your controller and define your current network by going to Settings / Networks / LAN. This is where you specify DHCP scope and settings. I did not do this and struggled to get DHCP running properly as a result. Be sure to also set NTP settings, as these will also be applied to your USG.

To configure your USG for adoption, hop on the 192.168.1.0/24 network and sign into 192.168.1.1 via a web browser. Username and password are both ubnt. On this screen you can specify WAN and LAN settings. Configure your USG to match the network and gateway settings you’ve defined in your controller and hit apply. Now you can go into your controller and adopt the firewall into your environment.

Firewall

Basic port forwarding rules, static routes, and firewall rules can all be handled in the controller GUI via settings / Routing & Firewall. The GUI assumes your gateway only has one public IP address going to it. If you have multiple public IPs then you will need to configure them in config.gateway.json (see the Advanced Configuration section below.)

DHCP

As stated in the Initial Setup section, this is handled by the controller. You can specify a DHCP scope in the USG’s limited web interface but any settings there are quickly overwritten by the controller pushing out its configuration.

DHCP reservations are handled in the controller via the clients tab (on the left.) Open the client you want to make a reservation for, click the settings cog (top right), click Network, then click “Use Fixed IP Address” and specify the IP you want that device to use.

You can also specify advanced DHCP settings under Settings / Services / DHCP.

Seeing active DHCP leases requires dropping to the CLI on the USG. SSH into the USG and run:

show dhcp leases

Traffic limiting

You can create User Groups in the Unifi interface which define maximum bandwidth usage. You can then assign that User group to a specific client in the Unifi interface.

NAT

The Unifi GUI only supports Destination NAT (DNAT) and only supports the gateway’s WAN IP. You can configure this via settings / Routing & Firewall / Port Forwarding. For more advanced configuration, see below.

Advanced Configuration

A major downside of the USG is that the Unifi interface, while awesome, is extremely limited when it comes to Firewall functions. Thus, most configuration has to be done in the command line to get it to compete with OPNSense.

The core concept with the Unifi ecosystem is that devices are controlled by the Unifi Network Management controller. Thus, with the USG, any changes made to the firewall itself are overwritten by the controller on next provision.

In order to persist any command line changes you make, you must create a config.gateway.json file as outlined here, then copy it to your controller, which will then push the config to your USG on each provision. You will run into problems if you get this json file wrong (reboot loops) so you want to be very sure everything is correct in that file. I recommend a json validator (or an IDE like VS Code.)

One good shortcut I’ve found when googling how to do things is to simply use “edgerouter” instead of “USG” for the search term. The syntax to configure the edgerouter is identical (they both run EdgeOS.)

The most foolproof way to get a config.gateway.json that works is to run the configure commands manually on your USG, then when everything is how you want it, run this command to generate the running config in json format:

mca-ctrl -t dump-cfg > config.txt

You can then read config.txt and look for the specific settings you configured and save them into your config.gateway.json. The JSON syntax follows the CLI commands, with each part of the command broken into different brackets and quotes. An example config.gateway.json looks like this:

{
  "service": {
    "nat": {
      "rule": {
        "4500": {
          "description": "port_forward_WAN2",
          "destination": {
            "address": "100.64.100.100",
            "port": "22"
          },
          "inbound-interface": "eth3",
          "inside-address": {
            "address": "192.168.1.100"
          },
          "protocol": "tcp",
          "type": "destination"
        }
      }
    }
  },
  "vpn": {
    "ipsec": {
      "site-to-site": {
        "peer": {
          "yyyy.ignorelist.com": {
            "authentication": {
              "id": "xxxx.ignorelist.com"
            },
            "local-address": "xxxx.ignorelist.com"
          }
        }
      }
    }
  }
}

DNS

Use the static-host-mapping parameter to specify static DNS entries. Make sure the fqdn is listed in your config, otherwise they may or may not work. Example snippet:

{
...
  "system": {
    "static-host-mapping": {
      "host-name": {
        "firewall": {
          "alias":[
            "firewall.jeppsonlocal"
          ],
          "inet": [
            "192.168.1.1"
          ]
        }
      }
    }
  }
...
}

Live traffic graphs

Sadly there is no live / realtime graphs in the UniFi interface. It’s still possible to get that information if you drop to CLI; however the utilities to see this are not installed by default – you will need to install them (iftop & bmon in my case.) Thanks to this helpful reddit post that got me going.

As of this writing the USG PRO 4 is based in Debian Wheezy, so you will need to add those repositories to the device in order to use apt-get to install iftop & bmon.

Be sure not to get the wrong Debian version. Also be sure not to issue apt-get upgrade – bad things will happen in both cases and you will need to hard reset your device to fix them.

You can add the repositories using the firewall configure command. These can be translated into a config.gateway.json if desired, but I decided not to since this is a pretty low level change that you might not want to happen on future devices. Also note that you will have to re-install these tools after a firmware upgrade.

configure
#Main wheezy archive
set system package repository wheezy components 'main contrib non-free'
set system package repository wheezy distribution wheezy
set system package repository wheezy url 'http://archive.debian.org/debian/'
commit
save
exit

sudo apt-get update
sudo apt-get install iftop bmon

If you want to undo the above changes, substitute set with delete:

#to remove:
configure
delete system package repository wheezy
commit

1:1 NAT

For 1:1 NAT you need 3 NAT rules (Destination NAT, Source NAT, and Hairpin NAT) and a corresponding firewall rule. Example:

{
    "service": {
        "nat": {
            "rule": {
                "1000": {
                    "description": "Mail 1:1 DNAT",
                    "destination": {
                        "address": "1.1.1.1",
                        "port": "25,80,443,465,587,993,995"
                    },
                    "inbound-interface": "pppoe0",
                    "inside-address": {
                        "address": "192.168.1.1"
                    },
                    "protocol": "tcp",
                    "type": "destination"
                },
                "3000": {
                    "description": "Mail 1:1 Hairpin NAT",
                    "destination": {
                        "address": "1.1.1.25",
                        "port": "25,80,443,465,587,993,995"
                    },
                    "inbound-interface": "eth0",
                    "inside-address": {
                        "address": "192.168.1.25"
                    },
                    "protocol": "tcp",
                    "type": "destination"
                },
                "5000": {
                    "description": "Mail 1:1 SNAT",
                    "type": "source",
                    "source": {
                        "address": "192.168.1.25"
                    }
                }
            }
        },
        "firewall": {
            "name": {
                "WAN_IN": {
                    "rule": {
                        "1000": {
                            "action": "accept",
                            "description": "Mail 1:1 DNAT",
                            "destination": {
                                "address": "192.168.1.25",
                                "port": "25,80,443,465,587,993,995"
                            },
                            "protocol": "tcp",
                            "log": "enable"
                        }
                    }
                }
            }
        }
    }
}

OpenVPN Site to Site

My OPNSense router had a site-to-site OpenVPN going with an OpenWRT router. Details on how to configure this are in a separate blog post here.


That covers the basics of what my OPNSense firewall was doing. It’s a bit of a learning curve but once I got past that it’s been working really well.

ProxMox VMs reboot when switch is rebooted

I came across an interesting situation where if I rebooted my Ubiquiti UniFi Switch 24 for a firmware upgrade, all my VM hosts would reboot themselves. It turned out to be due to my having enabled HA in ProxMox. The hosts would temporarily lose connectivity to each other and begin to fence themselves off from the cluster. This caused HA to kill the VMs on those hosts. Then once connectivity was restored everything would eventually come back up.

The proper way to fix this would be to have multiple paths for each host to talk to each other, so if one switch goes down the cluster is still able to communicate. In my case, where I only have one switch, the “poor man’s fix” was to simply disable HA altogether during the switch reboot, as outlined here. Then, once the switch is back up, re-enable HA.

On each node, stop the pve-ha-lrm service. Once it’s stopped on all hosts, stop the pve-ha-crm service. Then reboot your switch.

After the switch is back up, start pve-ha-lrm on each node first, then pve-ha-crm (if it doesn’t auto start itself) to re-enable HA.

OPENVPN site to site vpn between USG and openwrt

A new firewall means a new site to site VPN configuration. My current iteration of this is a USG Pro 4 serving as an OpenVPN server and a Netgear Nighthawk R8000 serving as a VPN client joining their two networks together.

First, I had to wrap my head around some concepts. To set this up you need three sets of certificates and a DH file:

  • CA: To generate and validate certificates
  • Server: To encrypt/decrypt traffic for the Server
  • Client: To encrypt/decrypt traffic from the Client
  • DH: Not a certificate but still needed by the server for encryption

The server and client will also need openvpn configurations containing matching encryption/hashing methods, CA public key, and protocol/port settings.

Generate certificates

If you already have PKI infrastructure in place you simply need to generate two sets of keys and a DH file for the server/client to use. If you don’t, the easy-rsa project comes to the rescue. This tutorial uses easy-rsa version 3.

I didn’t want to generate the certificates on my firewall so I picked a Debian system to do the certificate generation. First, install easy-rsa:

sudo apt install easy-rsa

In Debian easy-rsa is installed to /usr/share/easy-rsa/

Optional: Set desired variables by moving /usr/share/easy-rsa/vars.example to /usr/share/easy-rsa/vars and un-commenting / editing to suit your needs (in my case I like to extend the life of my certificates beyond two years.)

Next, create your PKI and generate CA certificates:

/usr/share/easy-rsa/easyrsa init-pki
/usr/share/easy-rsa/easyrsa build-ca

Now create your DH file. Grab a cup of coffee for this one, it can take up to ten minutes to complete:

/usr/share/easy-rsa/easyrsa gen-dh

Then create your server & client certificates. For this guide we are calling the server ovpn-server and the client ovpn-client

#For the server
/usr/share/easy-rsa/easyrsa gen-req ovpn-server nopass 
/usr/share/easy-rsa/easyrsa sign-req server ovpn-server

#For the client
/usr/share/easy-rsa/easyrsa gen-req ovpn-client nopass
/usr/share/easy-rsa/easyrsa sign-req client ovpn-client

You will be asked for a common name. Remember what you put here, you will need it later. If you just hit enter and accept the default the common name will match what was passed in the above commands (ovpn-server for the server certificate and ovpn-client for the client certificate.)

Lastly, copy these files to their respective hosts:

USG Server: CA, Server key & cert, and DH file. (substitute with IP of your device)

scp pki/dh.pem pki/ca.crt pki/private/ovpn-server.key  pki/issued/ovpn-server.crt admin@IP_OF_YOUR_USG:/config/auth/

OpenWRT Client: Client key & cert, and CA cert:

scp pki/private/ovpn-client.key pki/issued/ovpn-client.crt pki/ca.crt root@IP_OF_YOUR_OPENWRT:/etc/config/

USG: VPN Server

Documentation for the EdgeRouter is much easier to find than for the USG. Since they use the same operating system I based this off of this guide from Logan Marchione for the EdgeRouter. SSH into your USG and issue the following, substituting the $variables with the values you desire for your network.

Explanation of variables:

VPN_SUBNET: Used for VPN communication. Must be different from both server and client subnets.
SERVER_SUBNET: Subnet on server side you wish to pass to client network
VPN_PORT: Change this to desired listening port for the OpenVPN server
REMOTE_SUBNET: Subnet on client side you wish to pass to server network
REMOTE_NETMASK: Netmask of client subnet
REMOTE_VPN_IP: Static IP you wish to give the client on the VPN subnet.
REMOTE_CERT_NAME: Common name given to client certificate generated previously.

Replace $variables below before pasting into USG terminal:

configure
#OpenVPN config
set interfaces openvpn vtun0
set interfaces openvpn vtun0 description "OpenVPN Site to Site"
set interfaces openvpn vtun0 mode server
set interfaces openvpn vtun0 encryption aes256
set interfaces openvpn vtun0 hash sha256
set interfaces openvpn vtun0 server subnet $VPN_SUBNET
set interfaces openvpn vtun0 server push-route $SERVER_SUBNET
set interfaces openvpn vtun0 tls ca-cert-file /config/auth/ca.crt
set interfaces openvpn vtun0 tls cert-file /config/auth/ovpn-client.crt
set interfaces openvpn vtun0 tls key-file /config/auth/ovpn-client.key
set interfaces openvpn vtun0 tls dh-file /config/auth/dh.pem
set interfaces openvpn vtun0 openvpn-option "--port $VPN_PORT"
set interfaces openvpn vtun0 openvpn-option --tls-server
set interfaces openvpn vtun0 openvpn-option "--comp-lzo yes"
set interfaces openvpn vtun0 openvpn-option --persist-key
set interfaces openvpn vtun0 openvpn-option --persist-tun
set interfaces openvpn vtun0 openvpn-option "--keepalive 10 120"
set interfaces openvpn vtun0 openvpn-option "--user nobody"
set interfaces openvpn vtun0 openvpn-option "--group nogroup"
set interfaces openvpn vtun0 openvpn-option "--route $REMOTE_SUBNET $REMOTE_NETMASK $REMOTE_VPN_IP"
set interfaces openvpn vtun0 server client $REMOTE_CERT_NAME ip $REMOTE_VPN_IP
set interfaces openvpn vtun0 server client $REMOTE_CERT_NAME subnet $REMOTE_SUBNET $REMOTE_NETMASK

#Firewall config
set firewall name WAN_LOCAL rule 50 action accept
set firewall name WAN_LOCAL rule 50 description "OpenVPN Site to Site"
set firewall name WAN_LOCAL rule 50 destination port $VPN_PORT
set firewall name WAN_LOCAL rule 50 log enable
set firewall name WAN_LOCAL rule 50 protocol udp
commit

If the code above commits successfully, the next step is to add the config to config.gateway.json. The USG’s config is managed by its Unifi controller, so for any of the changes made above to stick we must copy them to /usr/lib/unifi/data/sites/default/config.gateway.json on the controller (create the file if it doesn’t already exist.)

A quick shortcut is to run the mca-ctrl -t dump-cfg command, then parse out the parts you want to go into config.gateway.json as outlined in the UniFi documentation. For the lazy, here is the config.gateway.json generated from the above commands (be sure to modify $variables to suit your needs.)

{
  "firewall": {
    "WAN_LOCAL": {
      "rule": {
        "50": {
          "action": "accept",
          "description": "OpenVPN Site to Site",
          "destination": {
            "port": "$VPN_PORT"
          },
          "log": "enable",
          "protocol": "udp"
        }
      }
    }
  },
  "interfaces": {
    "openvpn": {
      "vtun0": {
        "description": "OpenVPN Site to Site",
        "encryption": "aes256",
        "hash": "sha256",
        "mode": "server",
        "openvpn-option": [
          "--port $VPN_PORT",
          "--tls-server",
          "--comp-lzo yes",
          "--persist-key",
          "--persist-tun",
          "--keepalive 10 120",
          "--user nobody",
          "--group nogroup",
          "--route $REMOTE_SUBNET $REMOTE_NETMASK $REMOTE_VPN_IP"
        ],
        "server": {
          "client": {
            "$REMOTE_CERT_NAME": {
              "ip": "$REMOTE_VPN_IP",
              "subnet": [
                "$REMOTE_SUBNET $REMOTE_NETMASK"
              ]
            }
          },
          "push-route": [
            "$SERVER_SUBNET"
          ],
          "subnet": "$VPN_SUBNET"
        },
        "tls": {
          "ca-cert-file": "/config/auth/ca.crt",
          "cert-file": "/config/auth/ovpn-client.crt",
          "dh-file": "/config/auth/dh.pem",
          "key-file": "/config/auth/ovpn-client.key"
        }
      }
    }
  }
}

OpenWRT: VPN client

Configuration is doable from the GUI but I found much easier with the command line. I got a lot of the configuration from this gist from braian87b

Install openvpn and the luci-app-openvpn packages:

opkg update
opkg install openvpn luci-app-openvpn

OpenVPN config files are located in /etc/config. In addition to the certificates we copied there earlier, we will also want to copy the openvpn client configuration to that directory.

Here is the config file matching the configuration generated above. Again, remember to replace $variables with your config matching what was generated above. Save it to /etc/config/site2site.conf

#/etc/config/site2site.conf
client
dev tun
proto udp
remote $DNS_OR_IP_OF_USG_OPENVPN_SERVER $VPN_PORT
cipher AES-256-CBC
auth SHA256
resolv-retry infinite
nobind
comp-lzo yes
persist-key
persist-tun
verb 3
ca /etc/config/ca.crt
cert /etc/config/ovpn-client.crt
key /etc/config/ovpn-client.key
remote-cert-tls server

With the openvpn config file, client certificate & key, and CA certificate we are ready to configure firewall rules and instruct the router to initiate the VPN connection.

# a new OpenVPN instance:
uci set openvpn.site2site=openvpn
uci set openvpn.site2site.enabled='1'
uci set openvpn.site2site.config='/etc/config/site2site.conf'

# a new network interface for tun:
uci set network.site2sitevpn=interface
uci set network.site2sitevpn.proto='none' #dhcp #none
uci set network.site2sitevpn.ifname='tun0'

# a new firewall zone (for VPN):
uci add firewall zone
uci set firewall.@zone[-1].name='vpn'
uci set firewall.@zone[-1].input='ACCEPT'
uci set firewall.@zone[-1].output='ACCEPT'
uci set firewall.@zone[-1].forward='ACCEPT'
uci set firewall.@zone[-1].masq='1'
uci set firewall.@zone[-1].mtu_fix='1'
uci add_list firewall.@zone[-1].network='site2sitevpn'

# enable forwarding from LAN to VPN:
uci add firewall forwarding
uci set firewall.@forwarding[-1].src='lan'
uci set firewall.@forwarding[-1].dest='vpn'

# Finally, you should commit UCI changes:
uci commit

Monitor VPN connection progress by using logread. If all goes well you will see the successful connection established message. If not, you’ll be able to get an idea of what’s wrong.

logread -f

If all goes well you’ll now have a bidirectional VPN between your two sites; however, traffic from the server’s subnet going directly to the client router itself (the OpenWRT device’s IP) will be considered as coming from the WAN interface and will be blocked. If you need to access the OpenWRT device directly from the USG’s subnet, you’ll need to add a firewall rule allowing it to do so:

uci add firewall rule
uci set firewall.@rule[-1]=rule
uci set firewall.@rule[-1].enabled='1'
uci set firewall.@rule[-1].target='ACCEPT'
uci set firewall.@rule[-1].src='wan'
uci set firewall.@rule[-1].name='Allow VPN to access router'
uci set firewall.@rule[-1].src_ip='$SERVER_SUBNET'
uci set firewall.@rule[-1].dest_ip='$INTERNALL_IP_OF_OPENWRT_ROUTER'
uci commit

Troubleshooting

One-sided VPN

I fought for some time with the fact that the VPN was established, but only traffic going from the Client network to the Server network would work. Traffic from the OpenVPN server subnet to the OpenVPN client subnet would simply hang and not work.

I finally found on the ubiquiti forums that this is due to default OpenVPN behavior of restricting traffic from the server subnet to the client subnet (see the OpenVPN how-to for more information.) The solution is to add lines in the server config informing it of the client network and to allow traffic to it. Below is an example USG config allowing informing it of remote subnet 192.168.230/24 and assigning the Client an IP of 10.0.76.253:

set interfaces openvpn vtun5 server client client1 ip 10.0.76.253
set interfaces openvpn vtun5 server client client1 subnet 192.168.230.0/24

VPN status stays “stopped” in OpenWRT

The best way to troubleshoot is to look at the logs in realtime. SSH to the OpenWRT router and run the command “logread -f” then try to initiate the connection again. The errors there will point you to the problem.

Automate USG config deploy with Ubiquiti API in Bash

I have a new Ubiquiti Unifi Security Gateway Pro 4 which is pretty neat; however, the Unifi web interface is pretty limited. Most advanced firewall functions must be configured outside of the GUI. One must create a .json file with the configuration they need, copy that file to the Unifi controller, and then force a provision of the gateway to get it to pick up the new config.

I wanted a way to automate this process but very frustratingly Ubiquiti hasn’t documented their Unifi Controller API. I had to resort to reverse engineering their API by using my browser’s developer console to figure out which API calls were needed to do what I wanted. I then took the API functions from https://dl.ui.com/unifi/5.10.25/unifi_sh_api (the current unifi controller software download link which has unifi_sh_api) and embedded them into a bash script. Thanks to this forum post for the information on how to do this.

This bash script copies the specified config file to the Unifi controller via SCP, then uses curl to issue the API call to tell the controller to force a provision to the device having the supplied mac address.

#!/bin/bash
# Written by Nick Jeppson 08/01/2019
# Inspired by posts made from ubiquiti forums: https://community.ui.com/questions/API/82a3a9c7-60da-4ec2-a4d1-cac68e86b53c
# API interface functions taken from unifi_sh_api shipped with controller version 5.10.25, https://dl.ui.com/unifi/5.10.25/unifi_sh_api
#
# This bash script copies the specified config file to the Unifi controller via SCP
# It then uses curl to issue an API call to tell the controller to force a provision to the device with the supplied mac address. 

#### BEGIN VARIABLES ####
#Fill out to match your environment

gateway_mac="12:34:56:78:90:ab" #MAC address of the gateway you wish to manage
config_file="your_config_file.json"   #Path to config file
unifi_server="unifi_server_name"         #Name/IP of unifi controller server
unifi_gateway_path="/usr/lib/unifi/data/sites/default/config.gateway.json"    #Path to config.gateway.json on the controller
ssh_user="root"                 #User to SSH to controller as
username="unifi_admin_username"             #Unifi username
password="unifi_admin_password" #Unifi password
baseurl="https://unifi_server_name:8443" #Unifi URL
site="default"                  #Unifi site the gateway resides in

#### END VARIABLES ####

#Copy updated config to controller
scp $config_file $ssh_user@$unifi_server:$unifi_gateway_path

#API interface functions
cookie=$(mktemp)
curl_cmd="curl --tlsv1 --silent --cookie ${cookie} --cookie-jar ${cookie} --insecure "
unifi_login() {
    # authenticate against unifi controller
    ${curl_cmd} --data "{\"username\":\"$username\", \"password\":\"$password\"}" $baseurl/api/login
}

unifi_logout() {
    # logout
    ${curl_cmd} $baseurl/logout
}

unifi_api() {
    if [ $# -lt 1 ] ; then
        echo "Usage: $0 <uri> [json]"
        echo "    uri example /stat/sta "
        return
    fi
    uri=$1
    shift
    [ "${uri:0:1}" != "/" ] && uri="/$uri"
    json="$@"
    [ "$json" = "" ] && json="{}"
    ${curl_cmd} --data "$json" $baseurl/api/s/$site$uri
}

#Trigger a provision
unifi_login 
unifi_api /cmd/devmgr {\"mac\": \"$gateway_mac\", \"cmd\": \"force-provision\"}
unifi_logout

No more manually clicking provision after manually editing the config file on the controller!

LDAP nested group membership query

I have a lot of applications at work which do not support Active Directory but instead rely on LDAP queries for granting user access. A problem we have is much of our access is granted to a security group (known as a ROLE) and users are granted to that single security group to get access to things. This allows easier access granting to new hires / transfers. The problem is it makes LDAP queries much more difficult. Things are further complicated by the fact that sometimes users are directly granted access to resources instead of going through their ROLE security group.

Nested LDAP group search

I spent a lot of time researching LDAP nested group queries. I now have a functional way of doing semi-nested LDAP group searches. The scenario: a user could be directly added to a security group granting access to a resource, or could be a member of a security group which has access to the resource. I want the LDAP group search string to account for both. I accomplish this by combining these two queries:

Nested group membership query

Search groups beginning with the name ROLE for a specific member, then return what that ROLE group has access to

(&(objectClass=group)(DisplayName=ROLE*)(member=FQDN_OF_USER_IN_QUESTION)(memberOf=*))

Individually added group query

Search for all groups a specified member is a member of

(&(objectClass=user)(sAMAccountName=USERNAME_OF_USER_IN_QUESTION)(memberOf=*))

I combine these two queries by separating them out with an OR operator (|)

Combined query

Return the group membership of the user in question, as well as the group membership of the group beginning with the name ROLE that the user is a member of

(|(&(objectClass=group)(DisplayName=ROLE*)(member=FQDN_OF_USER_IN_QUESTION)(memberOf=*))(&(objectClass=user)(sAMAccountName=USERNAME_OF_USER_IN_QUESTION)(memberOf=*)))

It has three main parts:

  • Begin with an or operator (|
  • Have a new group with an AND operator (&
    • This requires everything in this query to be true
  • Make a second group with an AND operator

This works for our organization because ROLE groups are not nested within themselves and each user can only have one ROLE group assigned to them.

This combined query allows me to not have to “flatten” security groups for LDAP-bound applications. It makes me so happy.

This was made possible by a flurry of stack overflow posts:

https://stackoverflow.com/questions/32829104/ldap-query-with-wildcard

https://stackoverflow.com/questions/9564120/using-wildcards-in-ldap-search-filters-queries

https://stackoverflow.com/questions/6195812/ldap-nested-group-membership

https://stackoverflow.com/questions/1032351/how-to-write-ldap-query-to-test-if-user-is-member-of-a-group/1032426

https://www.novell.com/coolsolutions/feature/16671.html

Update /etc/hosts with current IP for ProxMox

ProxMox virtual environment is a really nice package for managing KVM and container visualization. One quirk about it is you need to have an entry in /etc/hosts that points to your system’s IP address, not 127.0.0.1 or 127.0.1.1. I wrote a little script to grab the IP of your specified interface and add it to /etc/hosts automatically for you. You may download it here or see below:

#!/bin/bash
#A simple script to update /etc/hosts with your current IP address for use with ProxMox virtual environment
#Author: Nicholas Jeppson
#Date: 4/25/2018

###Edit these variables to your environment###
INTERFACE="enp4s0" #the interface that has the IP you want to update hosts for
DNS_SUFFIX=""
###End variables section###

#Variables you shouldn't have to change
IP=$(ip addr show $INTERFACE |egrep 'inet '| awk '{print $2}'| cut -d '/' -f1)
HOSTNAME=$(hostname)

#Use sed to add IP to first line in /etc/hosts
sed -i "1s/^/$IP $HOSTNAME $HOSTNAME$DNS_SUFFIX\n/" /etc/hosts

Site to Site VPN between OPNsense & OpenWRT with Tinc

I’m a real glutton for punishment. I decided to upgrade my parents’ router to OpenWRT. The upgrade went smoothly except for one thing: The VPN I had established between my firewall and theirs.

This was a big enough headache that I even ended up switching my firewall from pfSense to OPNsense (Something I had been contemplating doing for a while anyway) hoping it would make things easier. It didn’t. In the end I abandoned OpenVPN entirely and instead went with Tinc.

Tinc is cool because it’s full mesh peer-to-peer instead of the traditional client / server model. If your equipment supports it, I’d definitely choose it over OpenVPN, especially if multiple sites are involved. A basic rundown of its configuration can be found here.

I used this site as a reference for how to set up tinc.  Essentially you decide on a network name, create private & public keys for each host, and configure each host to connect to each other via a config file & folder structure.

Tinc general configuration

On each device create an /etc/tinc/<network name>/hosts directory structure

mkdir -p /etc/tinc/<network name>/hosts
tincd -n <network name> -K 4096

To configure TINC we need some additional configuration files inside the /etc/tinc/<network name> directory

  • tinc.conf
  • tinc-up (script for bringing up the interface)
  • hosts/<hostname> (one for each location)

tinc.conf can be as simple as this:

Name = <name of host>
ConnectTo = <name of other host> 
#Add each host with an additional ConnectTo line

There needs to be a corresponding file in the hosts directory for each host. Example host file:

Address = <External IP of host>
Subnet = <Subnet other host will share>
#Add more subnets with additional Subnet lines

The host file also need’s the host’s public key. Append it to the end of the file:

cat /etc/tinc/<network name>/rsa_key.pub >> /etc/tinc/<network name>/hosts/<host name>

It’s easiest to generate the host files on each respective host, then copy them to all the other hosts.

The last step is to create the tinc-up script

#!/bin/sh
ubus -t 15 wait_for network.interface.$INTERFACE
ip link set $INTERFACE up ip addr add 172.16.0.1/24 dev $INTERFACE

Modify the IP used on each host so they don’t overlap. The private network here is what’s used for inter-host communication.

Make the script executable:

chmod 755 /etc/tinc/<network name>/tinc-up

OPNSense specific configuration

I got this working through an enhanced tinc package for OPNsense located here.  I will copypasta the content from that site here for easier reference:

Installation

The version might change, adjust it if fetch fails

fetch https://raw.githubusercontent.com/EugenMayer/tinc-opnsense/master/dist/os-tincdcustom-latest.txz
pkg install os-tincdcustom-latest.txz

1. your network

  1. copy the /usr/local/etc/tinc/example folder to /usr/local/etc/tinc/yournetwork
  2. enter yournetwork into /usr/local/etc/tinc/nets.boot to let this network be started on boot
  3. create keypairs by runng tincd -n <yournetwork> -K

2. your network configuration and tun device

  1. Edit /usr/local/etc/tinc/yournetwork/tinc.conf set the server you want to connect to and how this server is to be named
  2. Edit /usr/local/etc/tinc/yournetwork/tinc-up and adjust the network/netbitmask

3. finally the host configuration

  1. enter the /usr/local/etc/tinc/yournetwork/hosts folder and rename the files according to what you have chosen for youservername and theotherservername – they must match!
  2. enter the public key of the “this server” you find under /usr/local/etc/tinc/yournetwork/ into the according thisservernamefile and adjust the subnet this server offers (or subnets)
  3. enter the public key of the “other server” into the according theotherservername file and adjust the subnet the other server offers (or subnets)

4. OPNsense Interface/Gateway/Route/FW configuration

Please see this answer for a brief description

  • You need to create a Gateway, which is configured to go through tinc0 with “dynamic” (do not enter an IP on Gateway field)
  • You need to add a route to <remote subnet> through this gateway
  • Add your tinc0 interface in the Interface section. You can configure a ipv4 address or you don’t, does not matter. If you do, use your tinc-up configured address. Doing this enabled you to create FW Rules for the Tinc interface – which we will need.
  • Add a firewall on the Tinc interface to allow communication to local & remote subnets
    • Alternatively, add a single rule for the Tinc interface to allow any/any access (lazy, less secure)
  • Don’t forget to create a firewall rule allowing the port you’ve configured tinc to run on access from the internet.

OpenWRT specific configuration

Openwrt follows the general tinc configuration exactly. Make the appropriate folders and config files in /etc/tinc/<network name>/ and then test your configuration:

tincd -n <network name>

Once connection is established and working:

Create interface for your VPN (network / interfaces / add new interface) Select the name of your tinc network name from the list.

Next bridge your VPN to the LAN by going to Network / Firewall and editing your LAN zone. Select your VPN interface created earlier from the list and hit save & apply.

Run on startup

I could not find clear documentation on getting this to work on startup. There is a startup script for tinc but it doesn’t appear to launch my tinc config. I ended up modifying /etc/init.d/tinc and adding these lines to the start() and stop() functions. You could also just write your own simple init script to accomplish this.

start() {
...
/usr/sbin/tincd -n <network name>
}

stop() {
...
kill `pidof tincd`
}

Troubleshooting

Tincdcustom service won’t start in OPNSense

Starting from the GUI just does nothing, starting from CLI reveals this unhelpful error:

configctl tincdcustom status

Error (1)

From the OPNSense docs I determined which command I can run to see exactly why. The command is located in this configd configuration file: /usr/local/opnsense/service/conf/actions.d/actions_tincdcustom.conf

command:/usr/local/etc/rc.d/tincdcustom start

Doing that command manually revealed what the problem was:

/usr/local/etc/rc.d/tincdcustom start

Please create /usr/local/etc/tinc/nets.boot.

I had skipped step 1.2 of the tincdcustom instalaltion guide:

enter yournetwork into /usr/local/etc/tinc/nets.boot to let this network be started on boot

Once I added a single word – the name of the network I want to start on bootup – to /usr/local/etc/tinc/nets.boot – the daemon started and worked properly.

Running Tinc in verbose mode

Coming from the tinc documentation, I ran tinc in verbose mode on both of my hosts to troubleshoot why a connection wasn’t happening. It was very helpful.

tincd -n netname -d5 -D

Edit 9/9/2018

I had issues with my tinc startup script not working on the openwrt side. I found here that stated you should add this line to the top of your tinc-up config:

ubus -t 15 wait_for network.interface.$INTERFACE

This solved the startup issue for me.

Fix no internet in pfSense OpenVPN

I came across an issue with pfSense where I had created an openVPN connection but it would not work with internet traffic. The VPN connection established fine and I could connect to all internal hosts, but all internet traffic simply didn’t work. I checked the firewall logs and there were no firewall denies.

After scratching my head for a while I came across this post which suggests it might be due to the fact that there was no outbound NAT defined for the VPN network I created.

I went to Firewall / NAT / Outbound and sure enough, there was no outbound NAT rule for my VPN network. I manually created it for my VPN network and voila! Internet over the VPN!

Fix pfSense OpenVPN unable to bind to VIP issue

A while ago I bought multiple static IP addresses from my ISP. I configured the IPs as Virtual IP addresses through Firewall / Virtual IPs. Everything was dandy.. until I tried to assign OpenVPN to listen on one of my new IP addresses. No matter what I tried I could only get it to work if it listened on my gateway IP.. none of my other static IP addresses would work. The GUI would let me save the configuration, but if I headed over to Status / Openvpn I would see the following:

[error] Unable to contact daemon Service not running?

Digging further in the logs by going to Status / System Logs and then selecting the OpenVPN tab revealed the following snippet:

Time Process PID Message
May 15 19:42:00 openvpn 73195 Exiting due to fatal error
May 15 19:42:00 openvpn 73195 TCP/UDP: Socket bind failed on local address [AF_INET]<redacted>:443: Can’t assign requested address

After much digging I finally stumbled upon this post in the pfsense threads. In it they mention that in the Firewall / Virtual IPs screen not to bind  (in the interface option) your IP addresses to the WAN interface, but rather to bind them to localhost. That did the trick!

Site to site VPN between pfSense and DD-WRT

I’ve been trying to establish a site-to-site VPN connection between my house and my parents’ for a couple years now. Each time I try I become frustrated and eventually give up. No longer! I’ve finally gotten a site to site VPN working between my pfSense router and my parents’ Netgear Nighthawk R8000 running DD-WRT v3.

It was quite the undertaking for me to get these two systems talking. I drew a lot of inspiration from here. In order to get this to work you need to keep these things in consideration:

  • Protocol, port, device type, encryption cipher, hash algorithm, TLS authentication, and certificate settings all need to match
  • VPN IP addresses need to be assigned to both the server and the client
  • Routes for each network need to be established on both devices
  • Firewalls need to be configured to allow traffic to/from each network through the VPN tunnel

I used the following settings to get things working between these two devices:

OpenVPN Server (pfSense)

If you haven’t already, generate a certificate authority and server certificate. Do this in System / Cert Manager and click Add. When generating the certificate make sure the Certificate Type is Server Certificate.

General

  • Server mode: Peer to Peer (SSL / TLS)
  • Protocol: UDP
  • Device mode: tap

Cryptographic settings

  • Enable authentication of TLS packets: checked
  • Automatically generate a shared TLS authentication key: checked
  • Peer certificate authority & Server Certificate:  Select appropriate CA / certificate from the dropdowns here
  • DH Parameter length: 2048
  • Encryption algorithm: AES-256-CBC
  • Auth digest algorithm: SHA256 (256 BIT)
  • Certificate depth: One (Client + Server)

Tunnel Settings

  • IPv4 Tunnel network: Enter a unique (not existing on either netwonk) network here in CIDR format, ex 10.1.1.0/24
  • Ipv4 Local Network(s): Enter the networks you would like the remote network to access
  • IPv4 Remote Network(s): Enter the networks you would like the local network to access
  • Compression: Enabled with Adaptive Compression

Advanced Configuration

Optional? I found that for some reason the routing table wasn’t properly populated with the remote network on my pfSense server. I added a custom option to take care of that:

route <remote network> <remote subnet mask> <remote VPN IP>

In my example it ended up being “route 192.168.98.0 255.255.255.0 10.54.98.2”

Key export

You will need to export the Certificate Authority certificate as well as the client certificate & private key files for use with dd-wrt. Do this by going to System / Cert Manager. There are little icons to the right of the certificates where you can click to download them.

Export the CA certificate as well as both the certificate and key from whatever was specified in the Server Certificate section from the above OpenVPN configuration.

OpenVPN client (dd-wrt)

Go to Services / VPN and look for the OpenVPN Client section

  • Start OpenVPN Client: Enable
  • Tunnel Device: TAP
  • Tunnel Protocol: UDP
  • Encryption cipher: AES-256 CBC
  • Hash Algorithm: SHA256
  • User Pass Authentication: Disable
  • Advanced Options: Enable
  • LZO Compression: Adaptive
  • NAT: Enable
  • Firewall protection: Disable
  • Bridge TAP to br0: Enable
  • TLS Auth Key: <Paste the contents of the “Key” section under pfSense’s Cryptographic settings area of the OpenVPN server configuration>
  • CA Cert: <Paste contents of downloaded CRT file from pfsense’s CA>
  • Public Client Cert: <Paste contents of downloaded CRT file from pfsense’s certificate section>
  • Private Client Key: <Paste contents of downloaded .key file from pfsense’s certificate section>

IN THEORY this should be all that you have to do. The tunnel should establish and traffic should flow between both networks. Sadly it wasn’t that simple for my setup.

Troubleshooting

One-way VPN

After setting up the tunnel I saw everything was connected but the traffic was unidirectional (remote could ping local network but not visa versa.)

On the pfsense router, I ran

netstat -r | grep 192.168

(change the grep to whatever your remote network is)

I noticed that there were no routes to the remote network. To fix this I appended a server option in the OpenVPN server config (adjust this to match your networks)

route 192.168.98.0 255.255.255.0 10.54.98.2

Blocked by iptables

Adding the route on the server helped but things still weren’t getting through. I enabled logging on the DD-WRT side, consoled into the router, and ran the following:

watch -n 1 "dmesg | grep 192.168.98"

This revealed a lot of dropped packets from my OpenVPN server’s network. After a lot of digging I found this forum post that suggested a couple custom iptables rules to allow traffic between the bridged network and the OpenVPN network (adjust interface names as necessary)

iptables -I FORWARD -i br0 -o tap1 -j ACCEPT 
iptables -I FORWARD -i tap1 -o br0 -j ACCEPT

This doesn’t survive a reboot, so you’ll want to enter those two commands in the Administration / Commands section of the dd-wrt web configuration and click “Save Firewall”

Success

Finally, after a custom routing rule on the pfsense side and a custom iptables rule on the dd-wrt side, two way VPN has been established!