Tag Archives: firewall

Wireguard one-way traffic on USG Pro 4 after dual WAN setup

I have a site-to-site VPN between my Ubiquiti USG Pro-4 and an OpenWRT device over wireguard . It’s worked great until I got a secondary WAN connection as a failover connection since my primary cable connection has been flaky lately.

When you introduce dual-WAN on Ubiquiti devices you have to manually configure everything since the GUI assumes only one WAN connection. I configured my manual DNAT (port forwards) for each interface successfully but struggled to figure out why suddenly my Wireguard VPN between my two sites only went one way (remote side could ping all hosts on local side, but not visa-versa.)

After some troubleshooting I realized the firewall itself could ping the remote subnet just fine, it just wasn’t allowing local hosts to do so. I couldn’t find anything in firewall logs. Eventually I came across this very helpful page from hackad.nu that helped me to solve my problem.

The solution was to add a Firewall Modify rule specifically for the eth0 interface (where all my LAN traffic is routed through) to allow the source address of the subnets I want to traverse the VPN, then apply that modifier to the LAN_IN firewall rule for that interface. I had to do it for any VLANs I wanted to be able to use the Wireguard tunnel as well (vifs of eth0, VLAN 50 in my case)

Here is the relevant config.gateway.json sections, namely “firewall” and “interfaces”:

{
    "firewall": {
        "modify": {
            "Wireguard": {
                "rule": {
                    "10": {
                        "action": "modify",
                        "description": "Allow Wireguard traffic",
                        "modify": {
                            "table": "10"
                        },
                        "source": {
                            "address": "10.1.0.0/16"
                        }
                    }
                }
            }
        },
        "interfaces": {
            "ethernet": {
                "eth0": {
                    "firewall": {
                        "in": {
                            "ipv6-name": "LANv6_IN",
                            "modify": "Wireguard",
                            "name": "LAN_IN"
                        }
                    },
                    "vif": {
                        "50": {
                            "firewall": {
                                "in": {
                                    "ipv6-name": "LANv6_IN",
                                    "modify": "Wireguard",
                                    "name": "LAN_IN"
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

This did the trick! Wireguard is working both directions again, this time with my dual WAN connections.

Podman no internet in container fix

I’ve started experimenting with CentOS 8 & Podman (a fork of Docker.) I ran into an issue where one of my containers needed internet access, but could not connect. After some digging I found this site which explains why:

I had to configure the firewall on the podman host to allow for IP masquerade:

sudo firewall-cmd --zone=public --add-masquerade --permanent

After running the above command, my container had internet access!

Migrating from OPNSense to Ubiquiti Unifi Secure Gateway

I love the Ubiquiti Unifi interface. The only thing missing in my environment was the gateway. I had no complaints with my OPNSense firewall, but that missing section on the Unifi controller homepage haunted me, so I took the plunge and got a Unifi Secure Gateway Pro 4.

Basic Configuration

Initial setup

Official documentation is pretty detailed. Before you install your USG you will want to go into your controller and define your current network by going to Settings / Networks / LAN. This is where you specify DHCP scope and settings. I did not do this and struggled to get DHCP running properly as a result. Be sure to also set NTP settings, as these will also be applied to your USG.

To configure your USG for adoption, hop on the 192.168.1.0/24 network and sign into 192.168.1.1 via a web browser. Username and password are both ubnt. On this screen you can specify WAN and LAN settings. Configure your USG to match the network and gateway settings you’ve defined in your controller and hit apply. Now you can go into your controller and adopt the firewall into your environment.

Firewall

Basic port forwarding rules, static routes, and firewall rules can all be handled in the controller GUI via settings / Routing & Firewall. The GUI assumes your gateway only has one public IP address going to it. If you have multiple public IPs then you will need to configure them in config.gateway.json (see the Advanced Configuration section below.)

DHCP

As stated in the Initial Setup section, this is handled by the controller. You can specify a DHCP scope in the USG’s limited web interface but any settings there are quickly overwritten by the controller pushing out its configuration.

DHCP reservations are handled in the controller via the clients tab (on the left.) Open the client you want to make a reservation for, click the settings cog (top right), click Network, then click “Use Fixed IP Address” and specify the IP you want that device to use.

You can also specify advanced DHCP settings under Settings / Services / DHCP.

Seeing active DHCP leases requires dropping to the CLI on the USG. SSH into the USG and run:

show dhcp leases

Traffic limiting

You can create User Groups in the Unifi interface which define maximum bandwidth usage. You can then assign that User group to a specific client in the Unifi interface.

NAT

The Unifi GUI only supports Destination NAT (DNAT) and only supports the gateway’s WAN IP. You can configure this via settings / Routing & Firewall / Port Forwarding. For more advanced configuration, see below.

Advanced Configuration

A major downside of the USG is that the Unifi interface, while awesome, is extremely limited when it comes to Firewall functions. Thus, most configuration has to be done in the command line to get it to compete with OPNSense.

The core concept with the Unifi ecosystem is that devices are controlled by the Unifi Network Management controller. Thus, with the USG, any changes made to the firewall itself are overwritten by the controller on next provision.

In order to persist any command line changes you make, you must create a config.gateway.json file as outlined here, then copy it to your controller, which will then push the config to your USG on each provision. You will run into problems if you get this json file wrong (reboot loops) so you want to be very sure everything is correct in that file. I recommend a json validator (or an IDE like VS Code.)

One good shortcut I’ve found when googling how to do things is to simply use “edgerouter” instead of “USG” for the search term. The syntax to configure the edgerouter is identical (they both run EdgeOS.)

The most foolproof way to get a config.gateway.json that works is to run the configure commands manually on your USG, then when everything is how you want it, run this command to generate the running config in json format:

mca-ctrl -t dump-cfg > config.txt

You can then read config.txt and look for the specific settings you configured and save them into your config.gateway.json. The JSON syntax follows the CLI commands, with each part of the command broken into different brackets and quotes. An example config.gateway.json looks like this:

{
  "service": {
    "nat": {
      "rule": {
        "4500": {
          "description": "port_forward_WAN2",
          "destination": {
            "address": "100.64.100.100",
            "port": "22"
          },
          "inbound-interface": "eth3",
          "inside-address": {
            "address": "192.168.1.100"
          },
          "protocol": "tcp",
          "type": "destination"
        }
      }
    }
  },
  "vpn": {
    "ipsec": {
      "site-to-site": {
        "peer": {
          "yyyy.ignorelist.com": {
            "authentication": {
              "id": "xxxx.ignorelist.com"
            },
            "local-address": "xxxx.ignorelist.com"
          }
        }
      }
    }
  }
}

DNS

Use the static-host-mapping parameter to specify static DNS entries. Make sure the fqdn is listed in your config, otherwise they may or may not work. Example snippet:

{
...
  "system": {
    "static-host-mapping": {
      "host-name": {
        "firewall": {
          "alias":[
            "firewall.jeppsonlocal"
          ],
          "inet": [
            "192.168.1.1"
          ]
        }
      }
    }
  }
...
}

Live traffic graphs

Sadly there is no live / realtime graphs in the UniFi interface. It’s still possible to get that information if you drop to CLI; however the utilities to see this are not installed by default – you will need to install them (iftop & bmon in my case.) Thanks to this helpful reddit post that got me going.

As of this writing the USG PRO 4 is based in Debian Wheezy, so you will need to add those repositories to the device in order to use apt-get to install iftop & bmon.

Be sure not to get the wrong Debian version. Also be sure not to issue apt-get upgrade – bad things will happen in both cases and you will need to hard reset your device to fix them.

You can add the repositories using the firewall configure command. These can be translated into a config.gateway.json if desired, but I decided not to since this is a pretty low level change that you might not want to happen on future devices. Also note that you will have to re-install these tools after a firmware upgrade.

configure
#Main wheezy archive
set system package repository wheezy components 'main contrib non-free'
set system package repository wheezy distribution wheezy
set system package repository wheezy url 'http://archive.debian.org/debian/'
commit
save
exit

sudo apt-get update
sudo apt-get install iftop bmon

If you want to undo the above changes, substitute set with delete:

#to remove:
configure
delete system package repository wheezy
commit

1:1 NAT

For 1:1 NAT you need 3 NAT rules (Destination NAT, Source NAT, and Hairpin NAT) and a corresponding firewall rule. Example:

{
    "service": {
        "nat": {
            "rule": {
                "1000": {
                    "description": "Mail 1:1 DNAT",
                    "destination": {
                        "address": "1.1.1.1",
                        "port": "25,80,443,465,587,993,995"
                    },
                    "inbound-interface": "pppoe0",
                    "inside-address": {
                        "address": "192.168.1.1"
                    },
                    "protocol": "tcp",
                    "type": "destination"
                },
                "3000": {
                    "description": "Mail 1:1 Hairpin NAT",
                    "destination": {
                        "address": "1.1.1.25",
                        "port": "25,80,443,465,587,993,995"
                    },
                    "inbound-interface": "eth0",
                    "inside-address": {
                        "address": "192.168.1.25"
                    },
                    "protocol": "tcp",
                    "type": "destination"
                },
                "5000": {
                    "description": "Mail 1:1 SNAT",
                    "type": "source",
                    "source": {
                        "address": "192.168.1.25"
                    }
                }
            }
        },
        "firewall": {
            "name": {
                "WAN_IN": {
                    "rule": {
                        "1000": {
                            "action": "accept",
                            "description": "Mail 1:1 DNAT",
                            "destination": {
                                "address": "192.168.1.25",
                                "port": "25,80,443,465,587,993,995"
                            },
                            "protocol": "tcp",
                            "log": "enable"
                        }
                    }
                }
            }
        }
    }
}

OpenVPN Site to Site

My OPNSense router had a site-to-site OpenVPN going with an OpenWRT router. Details on how to configure this are in a separate blog post here.


That covers the basics of what my OPNSense firewall was doing. It’s a bit of a learning curve but once I got past that it’s been working really well.

Automate USG config deploy with Ubiquiti API in Bash

I have a new Ubiquiti Unifi Security Gateway Pro 4 which is pretty neat; however, the Unifi web interface is pretty limited. Most advanced firewall functions must be configured outside of the GUI. One must create a .json file with the configuration they need, copy that file to the Unifi controller, and then force a provision of the gateway to get it to pick up the new config.

I wanted a way to automate this process but very frustratingly Ubiquiti hasn’t documented their Unifi Controller API. I had to resort to reverse engineering their API by using my browser’s developer console to figure out which API calls were needed to do what I wanted. I then took the API functions from https://dl.ui.com/unifi/5.10.25/unifi_sh_api (the current unifi controller software download link which has unifi_sh_api) and embedded them into a bash script. Thanks to this forum post for the information on how to do this.

This bash script copies the specified config file to the Unifi controller via SCP, then uses curl to issue the API call to tell the controller to force a provision to the device having the supplied mac address.

#!/bin/bash
# Written by Nick Jeppson 08/01/2019
# Inspired by posts made from ubiquiti forums: https://community.ui.com/questions/API/82a3a9c7-60da-4ec2-a4d1-cac68e86b53c
# API interface functions taken from unifi_sh_api shipped with controller version 5.10.25, https://dl.ui.com/unifi/5.10.25/unifi_sh_api
#
# This bash script copies the specified config file to the Unifi controller via SCP
# It then uses curl to issue an API call to tell the controller to force a provision to the device with the supplied mac address. 

#### BEGIN VARIABLES ####
#Fill out to match your environment

gateway_mac="12:34:56:78:90:ab" #MAC address of the gateway you wish to manage
config_file="your_config_file.json"   #Path to config file
unifi_server="unifi_server_name"         #Name/IP of unifi controller server
unifi_gateway_path="/usr/lib/unifi/data/sites/default/config.gateway.json"    #Path to config.gateway.json on the controller
ssh_user="root"                 #User to SSH to controller as
username="unifi_admin_username"             #Unifi username
password="unifi_admin_password" #Unifi password
baseurl="https://unifi_server_name:8443" #Unifi URL
site="default"                  #Unifi site the gateway resides in

#### END VARIABLES ####

#Copy updated config to controller
scp $config_file $ssh_user@$unifi_server:$unifi_gateway_path

#API interface functions
cookie=$(mktemp)
curl_cmd="curl --tlsv1 --silent --cookie ${cookie} --cookie-jar ${cookie} --insecure "
unifi_login() {
    # authenticate against unifi controller
    ${curl_cmd} --data "{\"username\":\"$username\", \"password\":\"$password\"}" $baseurl/api/login
}

unifi_logout() {
    # logout
    ${curl_cmd} $baseurl/logout
}

unifi_api() {
    if [ $# -lt 1 ] ; then
        echo "Usage: $0 <uri> [json]"
        echo "    uri example /stat/sta "
        return
    fi
    uri=$1
    shift
    [ "${uri:0:1}" != "/" ] && uri="/$uri"
    json="$@"
    [ "$json" = "" ] && json="{}"
    ${curl_cmd} --data "$json" $baseurl/api/s/$site$uri
}

#Trigger a provision
unifi_login 
unifi_api /cmd/devmgr {\"mac\": \"$gateway_mac\", \"cmd\": \"force-provision\"}
unifi_logout

No more manually clicking provision after manually editing the config file on the controller!

Backup your systems with urBackup

In addition to my ZFS snapshots I decided to implement a secondary backup system. I decided to land on urbackup for ease of use and, more importantly, it was easier to set up.

Server Install

Assuming a Cent-based system:

cd /etc/yum.repos.d/
sudo wget http://download.opensuse.org/repositories/home:uroni/CentOS_7/home:uroni.repo
sudo yum -y install urbackup-server
sudo systemctl enable urbackup-server
sudo systemctl start urbackup-server

Open up necessary ports for the server:

sudo firewall-cmd --add-port=55413-55415/tcp --permanent
sudo systemctl reload firewalld

By default urbackup listens on port 55414 for connections. You can change this to port 80 and/or 443 for HTTPS by installing nginx and having it proxy the connections for you.

sudo yum -y install nginx
sudo systemctl enable nginx
sudo setsebool -P httpd_can_network_connect 1 #if you're using selinux

Copy the following into /etc/nginx/conf.d/urbackup.conf (make sure to change server_name to suit your needs)

server {
        server_name backup;

        location / {
                proxy_pass http://localhost:55414/;
        }
}

Then start nginx:

sudo systemctl start nginx

You should then be able to access the urbackup console by navigating to the IP / hostname of your backup server in a browser.

Client Install:

Urbackup can use a snapshot system known as dattobd. You should use it if you can in order to get more consistent backups, otherwise urbackup will simply copy files from the host which isn’t always desirable (databases, for example)

Install dattobd (optional):

sudo yum -y update
# reboot if your kernel ends up being updated
sudo yum -y localinstall https://cpkg.datto.com/datto-rpm/repoconfig/datto-el-rpm-release-$(rpm -E %rhel)-latest.noarch.rpm
sudo yum -y install dkms-dattobd dattobd-utils

Install urbackup client:

TF=`mktemp` && wget "https://hndl.urbackup.org/Client/2.1.15/UrBackup%20Client%20Linux%202.1.15.sh" -O $TF && sudo sh $TF; rm $TF
#Select dattobd when prompted if desired

Configure Firewall:

sudo firewall-cmd --add-port=35621-35623/tcp --permanent
sudo systemctl reload firewalld

Once a client is installed, assuming they’re on the same network as the backup server, they will automatically add themselves and begin backing up. If they don’t show up it’s usually a firewall issue.

Restore

Restoration of individual files is easily done through the web console. If you have a windows system, restoring from an image backup is also easy.

Linux hosts

Recovery is trickier if you want to restore a Linux system. Install an empty system of same distribution. Give it the same hostname. Install the client as outlined above, then run:

sudo /usr/local/bin/urbackupclientctl restore-start -b last

Troubleshooting

If for some reason the client not showing up after removing it from the GUI: Uninstall & re-install client software

sudo /usr/local/sbin/uninstall_urbackupclient
TF=`mktemp` && wget "https://hndl.urbackup.org/Client/2.1.15/UrBackup%20Client%20Linux%202.1.15.sh" -O $TF && sudo sh $TF; rm $TF

Migrate from Sophos UTM to pfSense part 1

I’ve been using a Sophos UTM virtual appliance as my main firewall / threat manager appliance for about two years now. I’ve had some strange issues with this solution off and on but for the most part it worked. The number of odd issues has begun to build, though.

Recently it decided to randomly drop some connections even though logs showed no dropped packets. The partial connections spanned across various networks and devices. I never did figure out what was wrong. After two days of furiously investigating (including disconnecting all devices from the network), the problem went away completely on its own with no action on my part. It was maddening – enough to drive me to pfSense.

As of version 2.2 pfSense can be fully virtualized in Xen, thanks to FreeBSD 10.1. This allowed me the option to migrate. Below are the initial steps I’ve taken to move to pfSense.

Features checklist

I am currently using the following functions in Sophos UTM. My goal is to move these functions to equivalents in pfSense:

  • Network firewall
  • Web Application Firewall, also known as a reverse proxy.
  • NTP server
  • PPPOE client
  • DHCP server
  • DNS server
  • Transparent proxy for content filtering and reporting
  • E-mail server / SPAM protection
  • Intrusion Detection system
  • Anti-virus
  • SOCKS proxy
  • Remote access portal (for downloading VPN configurations, etc)
  • Citrix Xenserver support (for live migration etc)
  • Log all events to a syslog server
  • VPN server
  • Daily / weekly / monthly e-mail reports on bandwidth usage, CPU, most visited sites, etc.

I haven’t migrated all of these function over to pfSense which is why this article is only Part 1. Here is what I have done so far.

Xenserver support

Installing xen tools is fairly straightforward thanks to this article. It’s simply a matter of dropping to a shell on your pfSense VM to install and enable xen tools

pkg install xe-guest-utilities
echo "xenguest_enable=\"YES\"" >> /etc/rc.conf.local
ln -s /usr/local/etc/rc.d/xenguest /usr/local/etc/rc.d/xenguest.sh 
service xenguest start

PPPoE client

The wizard works fine for configuring PPPOE, however I experienced some very strange issues with internet speed. Downstream would be fine but upstream would be incredibly slow. Another symptom was NAT / port forwarding appearing not to work at all.

It turns out the issue was pfSense’s virtualized status. There is a bug in the virtio driver that handles virtualized networking. You have to disable all hardware offloading on both the xenserver hypervisor and the pfSense VM to work around the bug. Details on how to do this can be found here. After that fix was implemented, speed and performance went back to normal.

DNS server

To get this working like it did in Sophos you have to disable the default DNS resolver service and enable the DNS forwarder service instead. Once DNS forwarder is enabled, check the box “register DHCP leases in DNS” so that DHCP hostnames come through to clients.

Syslog

Navigate to Status / system logs / settings tab and  tick “Send log messages to remote syslog server” and fill out the appropriate settings.

Note for Splunk users: the Technology Add-on for parsing pfsense logs expects the sourcetype to equal pfsense (not syslog). Create a manual input for logs coming from pfsense so it’s tagged as pfsense and not syslog (thanks to this post for the solution on how to get the TA to work properly.)

VPN

OpenVPN – wizard ran fine. Install OpenVPN Client Export utility package for easy exporting to clients. Once package is installed go to VPN / OpenVPN and you will see a new tab – Client Export.

Note you will need to create a user and check the “create certificate” checkbox or add a user certificate to existing user by going to System / User manager, Editing the user and clicking the plus next to User Certificates. The export utility will only show users that have valid certificates attached to them. If no users have valid certificates the Client Export tab will be blank.

Firewall

One useful setting to note is to enable NAT reflection. This allows you to access NATed resources as if you were outside the network, even though you are inside it. Do this by going to System / Advanced and clicking on the Firewall / NAT tab. Scroll halfway down to find the Network Address Translation section. Change NAT reflection mode for port forwards to Enable (Pure NAT)

It’s also very helpful to configure host and port aliases by going to Firewall / Aliases. This is roughly equivalent to creating Network and Host definitions in Sophos. When you write firewall rules you can simply use the alias instead of writing out hosts IPs and ports.

So far so good

This is the end of part 1. I’ve successfully moved the following services from Sophos UTM to pfSense:

  • Network firewall
  • PPPOE client
  • Log all events to a syslog server
  • VPN server
  • NTP server
  • DHCP server
  • DNS server
  • Xenserver support

I’m still working on moving the other services over. I’ve yet to find a viable alternative to the web application firewall but I haven’t given up yet.

Configuring a Cisco 881w from scratch

Below are my rough notes for configuring a Cisco 881W router with a broken WAN port from scratch. The commands below will get it running from a clean slate to a running system.

Enable saving of running config 

Create WAN VLAN

  • int vlan 5
  • name WAN

Enable DHCP on WAN port

  • int vlan 5
  • ip address dhcp
  • no shut

Configure port Fa3 to be the new WAN port

  • int fa3
  • switchport mode access
  • switchport access vlan 5

Create LAN VLAN

Assign remaining ports to Vlan 1

  • int range fa0 – 2
  • switchport mode access
  • switchport access vlan 1
  • no shut

Configure IP address for vlan 1

  • int vlan 1
  • ip address 192.168.1.1 255.255.255.0

Enable ssh

  • hostname mythbox
  • ip domain-name mooo.com
  • aaa new-model
  • username cisco password 0 cisco
  • crypto key generate rsa
  • ip ssh time-out 60
  • ip ssh authentication-retries 2

Configure an enable password

  • enable secret cisco

Create access list allowing only internal subnet for SSH (optional)

  • access-list 23 permit 192.168.1.0 0.0.0.255
  • line vty 0 4
  • access-class 23 in
  • transport input ssh

enable DHCP

  • service dhcp
  • no ip dhcp conflict logging
  • ip dhcp pool myth
  • network 192.168.1.0 /24
  • dns-server 192.168.1.1
  • default-router 192.168.1.1

Enable DNS server

  • ip dns server
  • TODO: Only allow DNS from internal interfaces

Enable PAT overload (NAT) from WAN interface

Configure outside / inside interfaces

  • int vlan 5
  • ip nat outside
  • int vlan 1
  • ip nat inside

Configure access list (config mode)

  • access-list 101 permit ip any any
    (can be more strict with first any by putting local subnet instead)
  • ip nat inside source list 101 interface vlan 5 overload

Disable VTP and STP

Needed if you’re going to use this in an enterprise network

  • no spanning-tree vlan 1
  • no spanning-tree vlan 5
  • vtp mode transparent

Configure Wireless access point

Place wireless on same VLAN as wired network

  • interface wlan-ap0
  • ip unnumbered vlan 1
  • no shut

Switch wireless access point to autonomous mode

  • conf t
  • service-module wlan-ap 0 bootimage autonomous
  • service-module wlan-ap 0 reload

Reset wireless module to default

Necessary if you don’t know the enable password of the current config of the wireless controller

  • service-module wlan-ap0 reset default-config

 Flash wireless module IOS version

Copy IOS image from USB drive to internal flash

  • copy usbflash:<filename> flash:<filename>

Configure TFTP server for the flash file

  • tftp-server flash:<filename>

Console into wireless module

  • ^Z
  • service-module wlan-ap0 session
  • password cisco/cisco if prompted (this is the router user/pass combination)
  • enable password is Cisco
  • escape character is ctrl shift 6 and then the letter x. router prompt will show. type disconnect to exit the wireless console shell

Flash wireless ROM via TFTP

  • archive download-sw overwrite /reload tftp://192.168.1.1/<filename>

Configure Wireless Access Point

Enable HTTP management

  • ip http server

Enable wireless radio

  • int Dot11Radio0
  • no shut

Connect to wireless management page

Determine the IP address of the wireless access point and connect to it using your browser

  • ^Z
  • sh ip interfaces brief

The default WAP page username/password is admin / Cisco

Manual Wireless Configuration

The wireless management page is very clunky. Following guidance from this site, I modified the steps slightly to include WPA 2 with AES encryption to allow for faster wireless N speeds. Replace ssid with desired wireless SSID and testpassword with desired password.

configure terminal
interface Dot11Radio0
encryption vlan 1 mode ciphers aes-ccm
ssid mythbox1
no shutdown
station-role root
exit

dot11 ssid mythbox1
Vlan 1
authentication open
authentication key-management wpa version 2
wpa-psk ascii 0 testpassword
guest-mode
exit

dot11 network-map

interface Dot11Radio0.1
encapsulation dot1Q 1 native
no ip route-cache
bridge-group 1
bridge-group 1 subscriber-loop-control
bridge-group 1 block-unknown-source
no bridge-group 1 source-learning
no bridge-group 1 unicast-flooding
bridge-group 1 spanning-disabled
exit

interface GigabitEthernet0.1
encapsulation dot1Q 1 native
no ip route-cache
bridge-group 1
no bridge-group 1 source-learning
bridge-group 1 spanning-disabled
exit