Category Archives: Web

Configure Zimbra live replication

I’ve recently configured live active replication from my Zimbra e-mail server to a backup server. This is really slick – in the event of primary server failure, I can bring up my secondary in a matter of minutes with no data loss. I used the Zimbra live sync scripts on Gitlab to accomplish this.

These are my notes on things I needed to do in addition to the readme to get things to work properly on my Zimbra 8.8.15 Open Source Edition installs on CentOS 7 boxes.

Install atd (at package):
sudo yum install atd

Make sure the backup server has the same firewall rules as the primary: https://wiki.zimbra.com/wiki/Ports

On the backup server, configure DNS for the mail server to resolve to the Backup server’s IP address. hostname: mail.server.dns -> mirror mail server.

Disable DNS forwarding for primary mail server domain if configured (to ensure mail goes to backup server in the event of switchover.)

Clone over prod mail server, spin up and change network settings:

  • keep hostname (important)
  • change IP, DNS, hosts to use new IP address/network

/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/hosts
/etc/resolv.conf

Ensure proper VLAN settings in backup VM (may be different than primary)

Systemd service:
add Environment=PATH=/opt/zimbra/bin:/opt/zimbra/common/lib/jvm/java/bin:/opt/zimbra/common/bin:/opt/zimbra/common/sbin:/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin
WorkingDirectory=/opt/zimbra

Remove start argument from ExecStart: ExecStart=/opt/zimbra/live_sync/live_syncd

This is the complete systemd unit for live sync:

[Unit]
Description=Zimbra live sync - to be run on the mirror server
After=network.target

[Service]
ExecStart=/opt/zimbra/live_sync/live_syncd
ExecStop=/opt/zimbra/live_sync/live_syncd kill
User=zimbra
Environment=PATH=/opt/zimbra/bin:/opt/zimbra/common/lib/jvm/java/bin:/opt/zimbra/common/bin:/opt/zimbra/common/sbin:/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin
WorkingDirectory=/opt/zimbra

[Install]
WantedBy=multi-user.target

Time limit

It looks like there’s a time limit for how long Zimbra keeps redo logs. It means you will get a lost mail situation if you try to bring your primary server back up after it’s been offline for too long (more than a few weeks.) If you’ve been failed over to your secondary mail server for more than two weeks, you’ll want to do the reverse procedure – clone the backup to the primary, edit IP addresses, then run the zimbra live sync. Log into the restored server to ensure mails from greater than 2 weeks ago are all there.

Fix cron output not being sent via e-mail

I had an issue where I had cron jobs that output data to stdout, yet mail of the output was never delivered. Everything showed fine in cron.log :

Aug  3 21:21:01 mail CROND[10426]: (nicholas) CMD (echo “test”)
Aug  3 21:21:01 mail CROND[10424]: (nicholas) CMDOUT (test)

yet no e-mail was sent. I finally found out how to fix this in a roundabout way. I came across this article on cpanel.net on how to silence cron e-mails. I then thought I’d try the reverse of a suggestion and add MAILTO= variable at the top of my cron file. It worked! Example crontab:

MAILTO=”youremail@address.com”
0 * * * * /home/nicholas/queue-check.sh

This came about due to my Zimbra box not sending system e-mails. In addition to the above, I had to configure zimbra as a sendmail alternative per this Zimbra wiki post: https://wiki.zimbra.com/wiki/How_to_%22fix%22_system%27s_sendmail_to_use_that_of_zimbra

Updating Zimbra to latest version

Recently a remote code execution bug came to light with Zimbra. It prompted me to update to the latest patch. I had some e-mail deliverability issues afterward. Here are my patch notes:

  • Download the latest version from https://www.zimbra.com/try/zimbra-collaboration-open-source/
    • Follow instructions as listed here: https://wiki.zimbra.com/wiki/Zimbra_Releases/8.8.15/P32#Redhat
  • Untar downloaded file, cd into directory and run ./install.sh as root
  • Re-install latest patches (I had frustrating 500 errors until I discovered this was the fix)
    • sudo yum reinstall zimbra-patch
  • Re-do any customization you’ve done to zimbra core
    • https://techblog.jeppson.org/2022/08/configure-zimbra-to-use-anymxrelay/
    • In my case, it was adding these lines to the smtp-amavis section:
    -o smtp_tls_security_level=none
    -o smtp_tls_wrappermode=no
  • Restart Zimbra services
    • sudo -u zimbra zmcontrol restart

Configure Zimbra to use AnyMXRelay

It turns out if you want to configure Zimbra to use an external SMTP relay service it can be a bit of a headache if that service doesn’t use port 25 or 587 to receive encrypted relay mail. Such is the case with AnyMXRelay. I decided to use AnyMXRelay to relay my mail since my Linode box keeps getting put on weird shadow blocklists despite mxtoolbox saying everything was fine.

It took some digging but I finally found this article on Zimbra’s wiki outlining what needs to happen. There are a few manual settings that need to be put in place on the command line in order to get this to work – namely, smtp_tls_wrappermode and smtp_tls_security_level.

In addition to the steps taken in this how-to for sending mail through a relay, you must also make these changes:

postconf -e smtp_tls_wrappermode=yes   # No Zimbra setting for smtp_tls_wrappermode yet
zmprov ms `zmhostname` zimbraMtaSmtpTlsSecurityLevel encrypt
zmprov ms `zmhostname` zimbraMtaSmtpTlsCAfile /opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt
zmprov ms `zmhostname` zimbraMtaSmtpSaslSecurityOptions noanonymous
zmprov ms `zmhostname` zimbraMtaSmtpSaslAuthEnable yes
zmprov ms `zmhostname` zimbraMtaSmtpSaslPasswordMaps lmdb:/opt/zimbra/conf/relay_password

Zimbra 8.5+ periodically applies settings automatically, so once you’ve made these changes, watch /var/log/zimbra.log for these lines

zmconfigd[25662]: Fetching All configs
zmconfigd[25662]: All configs fetched in 0.07 seconds
...
zmconfigd[25662]: All restarts completed in 1.80 sec

Once you see them, you can send some test mail. Tail /var/log/zimbra.log to see if it worked or to see any error messages.

If you get these error messages:

HANGUP after 0.08 from [IP]:56518 in tests before SMTP handshake
#or#
status=deferred (Cannot start TLS: handshake failure)

It means you must also add two configuration lines to the amavis configuration file in /opt/zimbra/common/conf/master.cf.in

-o smtp_tls_security_level=none
-o smtp_tls_wrappermode=no

So the complete section looks like this:

smtp-amavis unix -      -       n       -       %%zimbraAmavisMaxServers%%   smtp
    -o smtp_tls_security_level=none
    -o smtp_tls_wrappermode=no
    -o smtp_data_done_timeout=1200 
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
    -o max_use=20

Once you made the changes, save the file and restart all zimbra services with zmcontrol restart

The above disables TLS security for the antivirus piece. This could cause security issues if you Zimbra configuration is distributed to multiple hosts. In my case, this is an all-in-one server, so it does not matter.

Once I made the above changes, mail flowed through my external SMTP server successfully!

Self host postfix SMTP relay for Zimbra Mail Server

My notes for spinning up a small Debian linode server to act as an SMTP relay for my home network (note you will have to engage with linode support to enable mail ports for new accounts.)

Relay server configuration

Install postfix

sudo apt install postfix

Modify main.cf

sudo vim /etc/postfix/main.cf

Under TLS parameters, add TLS security to enable secure transfer of mail

smtp_tls_security_level = may
I decided not to open up postfix to the internet but instead my relay has a wireguard tunnel and postfix is allowed to relay only from that VPN subnet.

Add your subnets and relay restrictions further down:

mynetworks = 127.0.0.0/8 <YOUR_SERVER_SUBNET>
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated check_relay_domains
relay_domains = <MY_DOMAIN_NAME>
myhostname = <RELAYSERVER_HOSTNAME>
inet_interfaces = 127.0.0.1, <IP_OF_WIREGUARD_VPN_INTERFACE>

Zimbra configuration

In Zimmbra admin panel, edit your mail server

Configure / Servers / your_mail_server

MTA section

Add the DNS name and port of the relay system next to “Relay MTA for external deliverability”

If it won’t let you save, saying ::1 is required, you can add ::1 to MTA Trusted networks, however, on my Zimbra server this broke postfix. The symptoms were e-mails hanging and not sending. To fix, log into the Zimbra mail server and run as the zimbra user:

zmprov ms YOUR_MAIL_DOMAIN_NAME zimbraMtaMyNetworks ‘127.0.0.1 192.168.0.0/16’ (list of networks you had before but excluding ::1)

Then, issue postfix reload

That was it. A simple postfix SMTP relay which only accepts mail from my internal VPN (it doesn’t listen on the external interface at all.)

Troubleshooting

Realyed mail shows red unlock icon in Gmail (mail getting sent unencrypted)

Per postfix documentation I needed to enable secure transfer of mail by adding

smtp_tls_security_level = may

to main.cf

Mail does not send after adding ::1 to MTA Trusted Networks

Remove it via the CLI and reload postfix:

zmprov ms YOUR_MAIL_DOMAIN_NAME zimbraMtaMyNetworks '127.0.0.1 192.168.0.0/16' (list of networks you had before but excluding ::1)
postfix reload

Guacamole docker quick and easy

Apache Guacamole as an awesome HTML5 remote access gateway. Unfortunately it can be very frustrating to set up. They have docker images that are supposed to make the process easier, but I still ran into a lot of problems trying to get everything configured and linked.

Fortunately, a docker compose file exists to make Guacamole much easier to set up. Simply follow the instructions as laid out in the github readme:

  • Install docker & docker-compose
  • Clone their repository, run the initial prep script (for SSL keys & database initialization), and bring it up with docker-compose:
git clone "https://github.com/boschkundendienst/guacamole-docker-compose.git"
cd guacamole-docker-compose
sudo ./prepare.sh
sudo docker-compose up -d

Done! If you didn’t change anything in the docker-compose.yml file, you will have a new instance of Guacamole running on HTTPS port 8443 of your docker host. If you need to make changes (or if you forgot to run the prepare.sh file with sudo), you can run the reset.sh script which will destroy everything. You can then modify docker-compose.yml to suit your needs:

  • Whether to use nginx for HTTPS or just expose guacamole on port 8080 non-https (in case you already have a reverse proxy set up)
  • postgres password

Config files for each container are located within various folders in your guacamole-docker-compose folder. This can all be changed by editing the docker-compose.yml file.

Note this does configuration does not work with WOL, but as I do not use this feature I don’t mind.

Troubleshooting

docker ps will show running containers (docker ps -a shows all containers) If one is not running that should be, docker logs <container name> gives valuable insight as to why. In my case guacd was erroring out because I hadn’t initialized the database properly. Running the reset.sh script and starting over, this time running as sudo, did the trick.

Synchronize internet calendar to google calendar more frequently

Despite having my own e-mail server I still use Google Calendar for some things. I have an ICS file for the calendar for the Covid vaccination clinic I’m volunteering at. I ran into some frustrating sync problems when I tried to import it into my calendar. Google Calendar’s ICS sync process takes up to 12 hours, which was frustrating. I also had some mobile clients that wouldn’t even see the calendar imported from the ICS file.

I luckily found this post from Derek Antrican on stack exchange that outlines a script that you can configure to run at any given interval which will take all events in that ICS file and add/update/remove your calendar to match. It works beautifully. It’s a Google Apps script that you must copy into your own Google Scripts account to run.

First, go to the script here. Then go to Overview (i) and click “Make a Copy” in the top right (page icon.) Once the scripts are copied to your own script.google.com account, follow the instructions for configuring the script for your desired ICS URLs and other options, then click run.

My calendars are all synchronized and happy now.

Transcribe audio with Google Cloud speech-to-text api

I had a few audio files of an interview done with a late relative that I wanted to have Google transcribe for me. I wanted to supply an audio file and have it spit out the results. There are many ways to do this but I went with using the Google Cloud Platfrom speech-to-text API.

First I signed up for a GCP free trial via https://cloud.google.com/speech-to-text/ For my usage, it will remain free as 0-60 minutes of transcription per month is not charged: https://cloud.google.com/speech-to-text/pricing

Next, I needed to create GCP storage bucket as audio more than 10 minutes long cannot reliably be transcribed via the “uploading local file” option. I did this following the documentation at https://cloud.google.com/storage/docs/creating-buckets which walks you through going to their storage browser and creating a new bucket. From that screen I uploaded my audio files (FLAC in my case.)

Then I needed to create API credentials to use. I did this by going speech API console’s credentials tab and creating a service account, then saving the key to my working directory on my local computer.

Also on said computer I installed google-cloud-sdk (on Arch Linux in my case, it was as simple as yay -S google-cloud-sdk)

With service account json file downloaded & google-cloud-sdk installed I exported the GCP service account credentials into my BASH environment like so

export GOOGLE_APPLICATION_CREDENTIALS=NAME_OF_SERVICE_ACCOUNT_KEYFILE_DOWNLADED_EARLIER.json 

I created .json files following the format outlined in command line usage outlined in the quickstart documentation. I tweaked to add a line “model”: “video” to get the API to use the premium Video recognition set (as it was more accurate for this type of recording.) This is what my JSON file looked like:

{
  "config": {
      "encoding":"FLAC",
      "sampleRateHertz": 16000,
      "languageCode": "en-US",
      "enableWordTimeOffsets": false,
      "model": "video"

  },
  "audio": {
      "uri":"gs://googlestorarge-bucket-name/family-memories.flac"
  }
}

I then used CURL to send the transcription request to Google. This was my command:

curl -s -H "Content-Type: application/json" -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) https://speech.googleapis.com/v1/speech:longrunningrecognize -d @JSON_FILE_CREATED_ABOVE.json

If all goes well you will get something like this in response:

{
  "name": "4663803355627080910"
}

You can check the status of the transcription, which usually takes half the length of the audio file to do, by running this command:

curl -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) -H "Content-Type: application/json; charset=utf-8" "https://speech.googleapis.com/v1/operations/ID_NUMBER_ACQUIRED_ABOVE"

You will either get a percent progress, or if it’s done, the output of the transcription.

Success! It took some time to figure out but was still much better than manually transcribing the audio by hand.

Zimbra expired ldap certificate fix

I started getting SSL errors with my Zimbra mail server despite having a valid SSL certificate everywhere I knew where to check. When I tried to use zmcontrol status I got this error:

Unable to start TLS: SSL connect attempt failed error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed when connecting to ldap master.

Eventually I found this blog post explaining the problem – it’s with the LDAP component in Zimbra. You have to switch it from ldap to ldaps. Why did this change? I do not know.

ZIMBRA_HOSTNAME=_your_mail_server_dns_hostname_

sudo -u zimbra bash
zmlocalconfig -e ldap_master_url=ldaps://$ZIMBRA_HOSTNAME:636
zmlocalconfig -e ldap_url=ldaps://$ZIMBRA_HOSTNAME:636
zmlocalconfig -e ldap_starttls_supported=0
zmlocalconfig -e ldap_port=636
zmcontrol stop
zmcontrol start

This did the trick. The errors went away.

apache reverse proxy with basic authentication

I have an old Apache server that’s serving as a reverse proxy for my webcam. I swapped webcams out and unfortunately the new one requires authentication. I had to figure out how to get Apache to reverse proxy with the proper authentication. The best information I found was given by user ThR37 at superuser.com

Essentially you have to use an Apache module called headers to add an HTTP header to the request. On my Debian system this was not enabled, so I had to install it (thanks to Andy over at serverfault)

sudo a2enmod headers
#if you're on ubuntu then it's mod_headers

I then needed to generate the basic authentication hash for the header to pass. This was done via a simple python script:

#replace USERNAME:PASSWORD below with your credentials
import base64
hash = base64.b64encode(b'USERNAME:PASSWORD')
print hash

Save the above script into a file hash.py and then run it by typing

python hash.py

With headers enabled and hash acquired I just needed to tweak my config by adding a RequestHeader line:

RequestHeader set Authorization "Basic <HASH>"
#Replace <HASH> with hash acquired above

After adding that one line and restarting apache, it worked!