All posts by nicholas

Updating Zimbra to latest version

Recently a remote code execution bug came to light with Zimbra. It prompted me to update to the latest patch. I had some e-mail deliverability issues afterward. Here are my patch notes:

  • Download the latest version from https://www.zimbra.com/try/zimbra-collaboration-open-source/
    • Follow instructions as listed here: https://wiki.zimbra.com/wiki/Zimbra_Releases/8.8.15/P32#Redhat
  • Untar downloaded file, cd into directory and run ./install.sh as root
  • Re-install latest patches (I had frustrating 500 errors until I discovered this was the fix)
    • sudo yum reinstall zimbra-patch
  • Re-do any customization you’ve done to zimbra core
    • https://techblog.jeppson.org/2022/08/configure-zimbra-to-use-anymxrelay/
    • In my case, it was adding these lines to the smtp-amavis section:
    -o smtp_tls_security_level=none
    -o smtp_tls_wrappermode=no
  • Restart Zimbra services
    • sudo -u zimbra zmcontrol restart

Configure Zimbra to use AnyMXRelay

It turns out if you want to configure Zimbra to use an external SMTP relay service it can be a bit of a headache if that service doesn’t use port 25 or 587 to receive encrypted relay mail. Such is the case with AnyMXRelay. I decided to use AnyMXRelay to relay my mail since my Linode box keeps getting put on weird shadow blocklists despite mxtoolbox saying everything was fine.

It took some digging but I finally found this article on Zimbra’s wiki outlining what needs to happen. There are a few manual settings that need to be put in place on the command line in order to get this to work – namely, smtp_tls_wrappermode and smtp_tls_security_level.

In addition to the steps taken in this how-to for sending mail through a relay, you must also make these changes:

postconf -e smtp_tls_wrappermode=yes   # No Zimbra setting for smtp_tls_wrappermode yet
zmprov ms `zmhostname` zimbraMtaSmtpTlsSecurityLevel encrypt
zmprov ms `zmhostname` zimbraMtaSmtpTlsCAfile /opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt
zmprov ms `zmhostname` zimbraMtaSmtpSaslSecurityOptions noanonymous
zmprov ms `zmhostname` zimbraMtaSmtpSaslAuthEnable yes
zmprov ms `zmhostname` zimbraMtaSmtpSaslPasswordMaps lmdb:/opt/zimbra/conf/relay_password

Zimbra 8.5+ periodically applies settings automatically, so once you’ve made these changes, watch /var/log/zimbra.log for these lines

zmconfigd[25662]: Fetching All configs
zmconfigd[25662]: All configs fetched in 0.07 seconds
...
zmconfigd[25662]: All restarts completed in 1.80 sec

Once you see them, you can send some test mail. Tail /var/log/zimbra.log to see if it worked or to see any error messages.

If you get these error messages:

HANGUP after 0.08 from [IP]:56518 in tests before SMTP handshake
#or#
status=deferred (Cannot start TLS: handshake failure)

It means you must also add two configuration lines to the amavis configuration file in /opt/zimbra/common/conf/master.cf.in

-o smtp_tls_security_level=none
-o smtp_tls_wrappermode=no

So the complete section looks like this:

smtp-amavis unix -      -       n       -       %%zimbraAmavisMaxServers%%   smtp
    -o smtp_tls_security_level=none
    -o smtp_tls_wrappermode=no
    -o smtp_data_done_timeout=1200 
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20

Once you made the changes, save the file and restart all zimbra services with zmcontrol restart

The above disables TLS security for the antivirus piece. This could cause security issues if you Zimbra configuration is distributed to multiple hosts. In my case, this is an all-in-one server, so it does not matter.

Once I made the above changes, mail flowed through my external SMTP server successfully!

OpenWRT Wireguard client

My notes on how to configure an OpenWRT device to be a wireguard client (site to site VPN)

More or less follow the instructions from https://openwrt.org/docs/guide-user/services/vpn/wireguard/client

I commented out the IPv6 stuff as well as the pre-shared key. I also had already defined firewall rules so I skipped that section.

One note: make sure your WG_ADDR has the proper subnet mask (I made the mistake of making it a /32 when it needed to be a /24)


# Configuration parameters
WG_IF="wg0"
WG_SERV="remote_wireguard_server_address"
WG_PORT="remote_wireguard_port"
WG_ADDR="wireguard_subnet/wireguard_subnet_mask (/24 for example)"
#WG_ADDR6="fdf1:e8a1:8d3f:9::2/64"

# Generate keys
#umask go=
#wg genkey | tee wgserver.key | wg pubkey > wgserver.pub
#wg genkey | tee wgclient.key | wg pubkey > wgclient.pub
#wg genpsk > wgclient.psk
 
# Client private key
WG_KEY="$(cat wgclient.key)"
 
# Pre-shared key
#WG_PSK="$(cat wgclient.psk)"
 
# Server public key
WG_PUB="public_key_of_wireguard_server"

# Configure network
uci -q delete network.${WG_IF}
uci set network.${WG_IF}="interface"
uci set network.${WG_IF}.proto="wireguard"
uci set network.${WG_IF}.private_key="${WG_KEY}"
uci add_list network.${WG_IF}.addresses="${WG_ADDR}"
#uci add_list network.${WG_IF}.addresses="${WG_ADDR6}"
 
# Add VPN peers
uci -q delete network.wgserver
uci set network.wgserver="wireguard_${WG_IF}"
uci set network.wgserver.public_key="${WG_PUB}"
#uci set network.wgserver.preshared_key="${WG_PSK}"
uci set network.wgserver.endpoint_host="${WG_SERV}"
uci set network.wgserver.endpoint_port="${WG_PORT}"
uci set network.wgserver.route_allowed_ips="1"
uci set network.wgserver.persistent_keepalive="25"
uci add_list network.wgserver.allowed_ips="10.137.50.0/24"
#uci add_list network.wgserver.allowed_ips="::/0"
uci commit network
/etc/init.d/network restart

Add multiple search domains in CentOS 7

I needed to add multiple domains to search DNS with on my Cent7 box. It turns out there are two ways to do it. Cent7 uses networkmanager, so you can use the cli tool to add what you want, or you can edit the file directly.

Using nmcli:

sudo nmcli con mod eth0 ipv4.dns-search "domain1.org,domain2.org,domain3.org"

This causes nmcli to add this line to your network interface config file (/etc/sysconfig/network-scripts/ifcfg-eth0 in my case)

DOMAIN="domain1.org domain2.org domain3.org"

After either using nmcli or manually editing your file, simply restart the network service and your search domains now work!

Select all matches in VS Code

I came across a need to select all matches in VS Code. This was the procedure, thanks to stack overflow. I altered them for Mac, since that’s what I was using. For Windows/Linux, substitute CMD with CTRL

  1. CMD + F
  2. Type your search string
  3. CMD + SHIFT + L to select all occurrences found (max. 999)
  4. ESC (or close search dialog with top-right X)
  5. CMD + C
  6. Open new file
  7. CMD + V

Works great! It’s a way for me to easily get ONLY my search string from a file and discard the rest.

Wireguard on a USG Pro 4

I ran into some issues when trying to configure an OpenVPN tunnel between my Ubiquiti USG Pro 4 and a Debian VPS. I was very disappointed to discover that the version of OpenVPN on the USG only supports TLS 1.0! My issue was the Debian side rejecting that as insecure.

Thankfully, it was fairly painless to get Wireguard configured on the USG Pro 4. I was hesitant to do so at first because I knew every time my USG was updated I would lose the wireguard pacakge. Fortunately that can be resolved by configuring a post-install script. Thanks to ilar.in and calypte.cc and this github gist for the steps on how to do so.

curl -O https://raw.githubusercontent.com/britannic/install-edgeos-packages/master/install-pkgs
sudo install -o root -g root -m 0755 install-pkgs /config/scripts/post-config.d/install-pkgs
  • Add wireguard DEB package to persistent storage that the script will look for:
sudo mkdir -p /config/data/install-packages
cd /config/data/install-packages
curl -fLSs https://github.com/WireGuard/wireguard-vyatta-ubnt/releases/download/1.0.20200729-1/ugw4-v1-v1.0.20200729-v1.0.20200513.deb
  • Generate a public/private keypair for USG use
cd /config/auth
umask 077
wg genkey > wg_private.key
wg pubkey < wg_private.key > wg_public.key
  • Generate config.gateway.json config to use wireguard
    "interfaces": {
 ...
        "wireguard": {
            "wg0": {
                "address": "<IP_OF_USG_ON_WG_CLIENT_SUBNET>",
                "listen-port": "<WG_LISTEN_PORT>",
                "peer": {
                    "<ENDPOINT_CLIENT_PUBLIC_KEY>": {
                        "allowed-ips": "0.0.0.0/0",
                        "endpoint":  "<ENDPOINT_CLIENT_ADDRESS>:<ENDPOINT_CLIENT_PORT>
"
                    }
                },
                "private-key": "/config/auth/priv.key",
                "route-allowed-ips": false
            },
            "wg1": {
                "address": "<IP_OF_USG_ON_WG_SERVER_SUBNET",
                "firewall": {
                    "in": {
                        "name": "LAN_IN"
                    },
                    "local": {
                        "name": "LAN_LOCAL"
                    },
                    "out": {
                        "name": "LAN_OUT"
                    }
                },
                "listen-port": "<USG_WG_SERVER_LISTEN_PORT>",
                "mtu": "1352",
                "peer": {
                    "<PUBLIC_KEY_OF_WG_CONNECTING_CLIENT": {
                        "allowed-ips": "<SUBNETS_ON_REMOTE_HOST>"
                    }
                },
                "private-key": "/config/auth/wg-server.priv",
                "route-allowed-ips": true
            }
        }
    },

I have two different wireguard interfaces configured – wg0 to be a client to another server, and wg1 to be a server accepting other clients (site-to-site VPN.)

If you want to have multiple peers defined on a single wireguard interface, encapsulate the peers with brackets like so:

"peer": [{
						"--pubkey--": {
							"allowed-ips": [
								"172.255.252.2/32"
							],
							"persistent-keepalive": 60
						}
					},
					{
						"--pubkey--": {
							"allowed-ips": [
								"172.255.252.3/32"
							],
							"persistent-keepalive": 60
						}
					}
				],

Test configuration first

Before committing your config.gateway.json code, test it line by line by SSHing into the USG-Pro 4 and entering config mode. Then type out your JSON lines one at a time, with each key being a new argument separated by a space. The first section above would look like this:

configure
edit interfaces
set wireguard wg0 address WIREGUARD_ADDRESS
set wireguard wg0 listen-port WG_LISTEN_PORT
set wireguard wg0 peer ENDPOINT_CLIENT_PUBLIC_KEY allowed-ips 0.0.0.0/0
set wireguard wg0 peer ENDPOINT_CLIENT_PUBLIC_KEY endpoint ENDPOINT_ADDRESS:ENDPOINT_PORT
set wireguard wg0 private-key /config/auth/priv.key
set wireguard wg0 route-allowed-ips false
commit

If the commit works without error, you can then drop out of the configure section and look at your wireguard config:

exit
sudo wg show

If all looks well, then copy your config.gateway.json to your controller and trigger a reprovision.

Verify after provisioning: 

sudo netstat -npl | grep <WIREGUARD_PORT> | grep udp

Troubleshooting

USG not connecting to changed endpoint address

If you change the address of the wireguard endpoint, USG pro will not connect to that new address. You have to delete and re-create the interface (thanks to https://github.com/Lochnair/vyatta-wireguard/issues/72#issuecomment-423840448 for the information)

Fix this by deleting the wireguard interface

admin@Firewall:~$ configure
[edit]
admin@Firewall# delete interfaces wireguard
[edit]
admin@Firewall# commit


Then reprovision by making a small change, force provision, then change back, and force another provision (annoying) or alternatively reboot firewall.

Wireguard shows established but ping doesn’t work

Example error:

From 10.99.13.1 icmp_seq=5 Destination Host Unreachable
ping: sendmsg: Required key not available

To figure out what’s going on, enable logging to kernel buffer (dmesg) Thanks to procustodibus.com for the info.

echo module wireguard +p > /sys/kernel/debug/dynamic_debug/control

With debug on, ping again and check kernel messages (dmesg)

[Tue Dec 21 22:16:12 2021] wireguard: wg0: No peer has allowed IPs matching 10.99.13.2

This showed I didn’t have my access control properly configured. Modify /etc/wireguard/wg0.conf on your client config and make sure your AllowedIPs are properly letting traffic through.

AllowedIPs = 10.99.13.0/24

USG not allowing connections

Clients unable to connect to USG despite having a good config. Double check your firewall rules. I had neglected to create a WAN LOCAL rule allowing UDP packets on my wireguard port. Once that was configured, handshakes completed successfully.

Firewalls can’t ping each other

I had an issue where the firewalls would pass traffic through, but they couldn’t ping each other. The solution was to add the VPN subnet you created to allowed-ips on both sides of the connection.

Site to site Wireguard VPN between OPNSense & Debian Linux server

I have a Debian linode box acting as a wireguard server. I wanted to join my opnsense firewall to it to allow devices behind it to access the box through the wireguard tunnel. It was not as straightforward as I had hoped, but thankfully I got it all working.

OPNSense side

Documentation link

Install wireguard via GUI

Install the os-wireguard package. Manually drop to the CLI and install the wireguard package as well:
sudo pkg install wireguard

Configure Local instance

  • Name and listen port can be random. Tunnel address is the subnet you wish to expose to the other end (the subnet you wish to have access to the tunnel.)
  • Leave everything else blank and hit save
  • Edit your new connection and copy the Public key, this will need to be sent to the Debian server

Configure Endpoint

  • Name: hostname of Debian server
  • Public Key: Public key of server (can be obtained by running wg show on the server)
  • Shared Secret: blank (unless you’ve configured it on the server)
  • Allowed IPs: IPs or subnets on the Debian server you wish to expose to the client side (the OPNSense box)
  • Endpoint address: DNS name of Debian server
  • Endpoint port: Port Debian wireguard instance is listening on

Enable the VPN

General tab / Enable wireguard checkbox and hit apply.

Debian side

Take down the tunnel

sudo wg-quick down wg0

Edit wireguard config to add peer

sudo vim /etc/wireguard/wg0.conf

[Peer]
PublicKey = <PUBLIC_KEY_YOU_COPIED_IN_LOCAL_INSTANCE_STEP>
AllowedIPs = <IPs or Subnets behind the OPNSense side you wish to be exposed to the Debian side> 

Restart wireguard

sudo wg-quick up wg0

Check connections

Example wg show output below with dummy IPs:

sudo wg show
interface: wg0
  public key: f+/J4JO0aL6kwOaudAvZVa1H2mDzR8Nh3Vfeqq+anF8=
  private key: (hidden)
  listening port: 12345

peer: TuUW7diXcWlaV97z3cQ1/92Zal2Pm9Qz/W2OMN+v20g=
  endpoint: 10.0.0.1:54137
  allowed ips: 10.0.0.2/32
  latest handshake: 17 seconds ago
  transfer: 5.14 KiB received, 3.81 KiB sent

peer: CZuC/+wxvzj9+TiGeyZtcT/lMGZnXsfSs/h5Jtw2VSE=
  endpoint: 8.8.8.8:12345
  allowed ips: 192.168.1.1/32
  latest handshake: 7 minutes, 8 seconds ago
  transfer: 5.89 MiB received, 952.20 MiB sent

The endpoint: line gets populated when a successful VPN connection is made. If it’s missing, the tunnel was not established.

Troubleshooting

OPNSense box

Nothing happens after saving information and enabling tunnel

Make sure latest wireguard package is installed

sudo pkg install wireguard

Get more log output by opening a shell on your OPNSense box and running

sudo /usr/local/etc/rc.d/wireguard start

In my case I was getting this interesting message

[!] Missing WireGuard kernel support (ifconfig: SIOCIFCREATE2: Invalid argument). Falling back to slow userspace implementation.
[#] wireguard-go wg0
┌──────────────────────────────────────────────────────┐
│                                                                                                                                                   │
│   Running wireguard-go is not required because this                                                     │
│   kernel has first class support for WireGuard. For                                                          │
│   information on installing the kernel module,                                                                 │
│   please visit                                                                                                                             │
│         https://www.wireguard.com/install/                                                                           │
└──────────────────────────────────────────────────────┘

I fixed this problem by manually installing wireguard with the pkg install command above.

Debian box

Wireguard config not saving

make sure to stop the tunnel first, otherwise your changes get overwritten by the daemon.

sudo wg-quick down wg0
<make changes>
sudo wg-quick up wg0

Self host postfix SMTP relay for Zimbra Mail Server

My notes for spinning up a small Debian linode server to act as an SMTP relay for my home network (note you will have to engage with linode support to enable mail ports for new accounts.)

Relay server configuration

Install postfix

sudo apt install postfix

Modify main.cf

sudo vim /etc/postfix/main.cf

Under TLS parameters, add TLS security to enable secure transfer of mail

smtp_tls_security_level = may
I decided not to open up postfix to the internet but instead my relay has a wireguard tunnel and postfix is allowed to relay only from that VPN subnet.

Add your subnets and relay restrictions further down:

mynetworks = 127.0.0.0/8 <YOUR_SERVER_SUBNET>
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated check_relay_domains
relay_domains = <MY_DOMAIN_NAME>
myhostname = <RELAYSERVER_HOSTNAME>
inet_interfaces = 127.0.0.1, <IP_OF_WIREGUARD_VPN_INTERFACE>

Zimbra configuration

In Zimmbra admin panel, edit your mail server

Configure / Servers / your_mail_server

MTA section

Add the DNS name and port of the relay system next to “Relay MTA for external deliverability”

If it won’t let you save, saying ::1 is required, you can add ::1 to MTA Trusted networks, however, on my Zimbra server this broke postfix. The symptoms were e-mails hanging and not sending. To fix, log into the Zimbra mail server and run as the zimbra user:

zmprov ms YOUR_MAIL_DOMAIN_NAME zimbraMtaMyNetworks ‘127.0.0.1 192.168.0.0/16’ (list of networks you had before but excluding ::1)

Then, issue postfix reload

That was it. A simple postfix SMTP relay which only accepts mail from my internal VPN (it doesn’t listen on the external interface at all.)

Troubleshooting

Realyed mail shows red unlock icon in Gmail (mail getting sent unencrypted)

Per postfix documentation I needed to enable secure transfer of mail by adding

smtp_tls_security_level = may

to main.cf

Mail does not send after adding ::1 to MTA Trusted Networks

Remove it via the CLI and reload postfix:

zmprov ms YOUR_MAIL_DOMAIN_NAME zimbraMtaMyNetworks '127.0.0.1 192.168.0.0/16' (list of networks you had before but excluding ::1)
postfix reload

Fix KVMD not starting after updating piKVM

piKVM is amazing. I had one controlling an old desktop of mine for over 18 months with no issues. I decided to update its software and ran into some problems.

The first problem was pacman was returning a 404 when trying to update. I guess in the last year and a half the repository URL had changed. I had to edit /etc/pacman.conf and update the URL:

[pikvm]
Server = https://files.pikvm.org/repos/arch/rpi4
SigLevel = Required DatabaseOptional

After fixing that, running pacman -Syu and answering yes, I rebooted, but found that kvmd would not start. The first symptom: HTTP 500 from nginx. Digging in I found that two services were failing to start: kvmd-tc358743.service and kvm-otg.service.

v4l2-ctl[429]: Cannot open device /dev/kvmd-video, exiting
kvmd-otg[398]: RuntimeError: Can't find any UDC

With these two services bailing the web UI wouldn’t start. I checked the kernel log and the tc358743 device was not detected at all. I was about to give up and just reflash the device when I noticed two files in /boot: cmdline.txt.pacsave and config.txt.pacsave. I know from my experience in arch that it means I had some configurations get clobbered. Running a diff between the two I found some very important lines omitted:

dtoverlay=tc358743
dtoverlay=disable-bt
dtoverlay=dwc2,dr_mode=peripheral

I restored the .pacsave files and rebooted, and it worked! Everything came back.

Next time I won’t wait so long between software updates.

piKVM pushover startup script

I’ve had an issue where I wasn’t sure if my dynamic DNS provider registered properly. I then realized that I have a piKVM attached to one of my servers that boots on powerup, even if the server does not. I could utilize this piKVM to help me out.

Thanks to inspiration from Chris Dzombak I was able to whip up a little script that runs on startup. This script waits 5 minutes to allow for my firewall and modem to boot up, then sends a pushover notification to let me know the piKVM is online and what its external IP address is.

To get it working on the piKVM I had to enter into RW mode, write and save the script, add execute permissions to the script, then configure a systemd service to run the script at startup.

Here is the script, saved under /root/boot-pushover.sh

#!/usr/bin/env bash
set -eu

#Wait 5 minutes to allow router bootup
sleep 300

TOKEN="PUSHOVER_APPLICATION_TOKEN"
USER="PUSHOVER_USER_TOKEN"
EXTERNAL_IP="$(curl ifconfig.me)"
MESSAGE="$(hostname) is online. External IP: $EXTERNAL_IP"

#Send pushover command to alert it's up and send its external IP
curl -s \
  --form-string "token=$TOKEN" \
  --form-string "user=$USER" \
  --form-string "message=$MESSAGE" \
  https://api.pushover.net/1/messages.json

Set executable: chmod +x /root/boot-pushover.sh

Here is the systemd service, saved under /etc/systemd/system/boot-pushover-notification.service

[Service]
Type=oneshot
ExecStart=/root/boot-pushover.sh
RemainAfterExit=yes
User=root
Group=root
RestartSec=15
Restart=on-failure

[Unit]
Wants=network.target
After=network.target nss-lookup.target

[Install]
WantedBy=multi-user.target

Reload daemons & enable startup:

systemctl daemon-reload
systemctl enable boot-pushover-notification.service

Test by exiting rw mode and rebooting the piKVM:

ro
reboot

It works really well!