Category Archives: CLI

CLI, Virtualization

Recover CEPH mon cluster after host failure

Leave a comment

I recently had a Proxmox node’s root disk fail. I was able to bring it back up fairly quickly with a new Proxmox install but struggled with ceph mon process for that node. I encountered an issue where the mon process ate up all the RAM on the host. I left the cluster in this state overnight and by the morning the mon cluster had lost its mind and read/write was broken for the entire CEPH cluster.

The fix was to stop mon processes on all nodes. Then pick a healthy node and manually bring it back up after manually removing all other nodes from its config. Then on every node, manually destroy and re-create the ceph mon configuration. Here are the links I used to help me figure it all out:

https://forum.proxmox.com/threads/ceph-cluster-can-not-start-monitors.34689

https://forum.proxmox.com/threads/ceph-cannot-create-monitor-monitor-address-xxx-already-in-use-500.130932

https://forum.proxmox.com/threads/ceph-how-to-delete-dead-monitor.61172

Rescue unhealthy mon cluster

Taken from https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#removing-monitors-from-an-unhealthy-cluster

Stop all ceph-mon daemons on all Monitor hosts:

ssh {mon-host}
systemctl stop ceph-mon.target

Repeat this step on every Monitor host.

Identify a surviving Monitor and log in to the Monitor’s host:

ssh {mon-host}

Extract a copy of the monmap file by running a command of the following form:

ceph-mon -i {mon-id} --extract-monmap {map-path}

Here is a more concrete example. In this example, hostname is the {mon-id} and /tmp/monmap is the {map-path}:

ceph-mon -i `hostname` --extract-monmap /tmp/monmap

Remove the non-surviving or otherwise problematic Monitors:

monmaptool {map-path} --rm {mon-id}

For example, suppose that there are three Monitors—mon.a, mon.b, and mon.c—and that only mon.a will survive:

monmaptool /tmp/monmap --rm b
monmaptool /tmp/monmap --rm c

Inject the surviving map that includes the removed Monitors into the monmap of the surviving Monitor(s):

ceph-mon -i {mon-id} --inject-monmap {map-path}

Continuing with the above example, inject a map into Monitor mon.a by running the following command:

ceph-mon -i a --inject-monmap /tmp/monmap

Start only the surviving Monitors.

Verify that the Monitors form a quorum by running the command ceph -s.

The data directory of the removed Monitors is in /var/lib/ceph/mon: either archive this data directory in a safe location or delete this data directory. However, do not delete it unless you are confident that the remaining Monitors are healthy and sufficiently redundant. Make sure that there is enough room for the live DB to expand and compact, and make sure that there is also room for an archived copy of the DB. The archived copy can be compressed.

Additional procedures I took to regain full health

Remove dead mons from ceph.conf

vi /etc/pve/ceph.conf

Remove them both from mon_host line and [mon.<hostname>] stanzas

Remove systemd unit

systemctl stop ceph-mon@<hostname>.service
rm /etc/systemd/system/ceph-mon.target.wants/ceph-mon@<hostname>.service
systemctl daemon-reload

Remove ceph mon files

rm -r /var/lib/ceph/mon/ceph-<hostname>/

Re-create mon process

pveceph mon create

Reboot host. I only had to do this on one stubborn node that still wouldn’t start the mon process. It came back up fine after reboot.

CLI, OS

CRON: Get full output only on error

Leave a comment

I found this very handy trick to cron jobs to make sure they only e-mail you on failure, and when they do, you get the entire output of the cronjob. Thanks to this serverfault.com post

The trick is to put the entire thing including STDOUT and STDERR in a subshell outputting to variable OUTPUT, and then append || echo “$OUTPUT” to the end of the command. If the command is successful, the echo half never fires, meaning silent execution. If it does error, then the full output is echoed, which will get picked up and e-mailed from cron (if configured correctly.)

The example I used is for paperless-ngx backups: 

OUTPUT=$(cd /mnt/docker/paperless && docker compose exec -T webserver document_exporter -p ../export/backup --delete 2>&1) || echo "$OUTPUT"

It works!

CLI, Web

Find SMTP Open Relays with SWAKS

Leave a comment

I recently discovered I had an open mail relay. This situation was particularly frustrating because tools like mxtoolbox kept reporting my mail server as not having an open relay. All the standard tools simply looked at failure to issue commands as evidence that relay access was denied, when in fact that was NOT the case!

The clue that the “Not an open relay” message was a lie was in the details. Instead of a message about “access denied” the tool returned 530 5.7.0 Must issue a STARTTLS command first [205 ms]

I was only able to finally get confirmation about my open mail relay from SWAKS. It really is a wonderful tool. This is the command that finally cracked the case for me:

swaks --to user@domain.com --server mail.domain.com   --from support@domain.com --port 587 -tls --tls-protocol tlsv1_2

This resulted in a successful unauthenticated e-mail sent from my domain. The open relay was used TLS on port 587, which was opened to allow end users to be able to send mail from their devices. This was made possible thanks to an SSH tunnel I had set up to send port 587 on the external IP directly to the internal mail server, not realizing this would treat all traffic to that IP as trusted.

I’ve closed all that up to prevent any more incursions. It is quite interesting that I had that open for quite some time before someone decided to try to relay using TLS on port 587, and that none of the standard web tools reported it as an open relay!

CLI

Pipe mysql export from a docker container into a different docker container

Leave a comment

I recently needed to take a mysql database export from one docker container and import it into a different docker container. It took a while to get the commands I needed, with this stackoverflow thread helping me to understand what needed to be done.

I initially tried to use docker exec -it <container> /bin/bash but when I tried to pipe a mysqldump to a mysql to another container I kept getting cannot attach stdin to a TTY-enabled container because stdin is not a terminal

I tracked that down to the -t option of docker exec. Once I removed that I was able to pipe the dump successfully!

sudo docker exec -i <SOURE_MYSQL_CONTAINER> mysqldump -u <USER> -p<PASSWORD> <DATABASE_TO_DUMP> | sudo docker exec -i <DESTINATION_MYSQL_CONTAINER> mysql -u<DESTINATION_DATABASE_USER> -p<DESTINATION_DATABASE_PASSWORD> <DESTINATION_DATABASE>
CLI

Get a summary of disk usage from select files with find, sed, du, and xargs

Leave a comment

I wanted a quick way in the command line to get the disk usage of a bunch of zip files I downloaded in the previous day. I also wanted them sorted by filename and to have quotes surround each filename. I learned from this stackexchange post that du -ch is the command I want to accomplish this. Here is my final command. It works! Note: I ran this on a mac, so I had to use gsed because the version of sed that ships with mac is rather crippled. On linux the command would simply be sed instead of gsed

find . -name "*.zip" -mtime -1|sort -h|sed 's/.\//"/g'|sed 's/.zip/.zip"/g'|gsed -z 's/\n/ /g'|xargs du -ch

The output looks like this (snippet – not the full output):

753M V-A – Mixed by Mahiane – OXYCANTA.zip
912M V-A – Mixed by Nova – ALBEDO.zip
816M V-A – Selected by Fishimself – AMBROSIA (24bits).zip
977M Various Artists – FAHRENHEIT PROJECT – Part 1.zip
992M Various Artists – FAHRENHEIT PROJECT – Part 2.zip
848M Various Artists – FAHRENHEIT PROJECT – Part 3.zip
849M Various Artists – FAHRENHEIT PROJECT – Part 4.zip
817M Various Artists – FAHRENHEIT PROJECT – Part 5.zip
897M Various Artists – FAHRENHEIT PROJECT – Part 6.zip
897M Various Artists – FAHRENHEIT PROJECT – Part 7.zip
737M Various Artists – ISOLATED (24bit).zip
817M Various Artists – OPIA (24bit).zip
55G total

For the curious, I had purchased the Ultimae Digital Collection. Great stuff.

CLI, Networking

Port Forward from Internet to Wireguard interface

1 Comment

I needed to give my CGNAT-backed home internet a way to have a public IP address. My first solution was to use wireguard directly, and forward ports as needed. I came across this article that helped me do it. The key was to enable packed masquerading so the return path could be completed. Example wireguard server config:

# packet forwarding
PreUp = sysctl -w net.ipv4.ip_forward=1

# port forwarding
PreUp = iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 2000 -j DNAT --to-destination 10.0.0.1:8080
PostDown = iptables -t nat -D PREROUTING -i eth0 -p tcp --dport 2000 -j DNAT --to-destination 10.0.0.1:8080

# packet masquerading
PreUp = iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
PostDown = iptables -t nat -D POSTROUTING -o wg0 -j MASQUERADE

Example wireguard client config:

PreUp = iptables -t nat -A POSTROUTING -o wg0
PostUp = iptables -A FORWARD -i %i -j ACCEPT
PostDown = iptables -D FORWARD -i %i -j ACCEPT
PostDown = iptables -t nat -D POSTROUTING -o wg0

Make sure you have correct allowedIPs configured on client and server. This does work, but it shows the source IP as being the VPN destination. If you value seeing what true external source IPs are, then this solution is not for you (eg seeing external IPs accessing a webserver.)

DNS resolution inside docker containers

I had an issue where docker containers weren’t resolving DNS properly over this VPN tunnel. I found this site that explained I needed to update my docker daemon.json to explicitly specify which DNS servers to use, then restart docker:

{
  "dns": ["172.17.0.1","10.10.10.1"]
}
CLI

Increment modified date of files in a directory based on first file

Leave a comment

I had an issue in Immich where it was sorting pictures by their modified date. The modified dates are random, but the filenames are not. I wanted the album to sort by filename, and to do that I needed to get each filename to have a modified time in the same order. This was my solution (run within the directory in question) :

date=$(date -r $(ls | head -1) +%s); for file in *.jpg; do touch -m -d "@$date" $file; ((date+=1)); done

This bash one-liner does the following:

  • Sets a date variable by taking the modified date of the first file in the directory and converting it to epoch time
  • Goes through each JPG file in the directory and executes a touch command to set the date of that file to the date variable
  • Increments the date variable by 1 before processing the next file

The end result is now the order the files are in by modified date match their filename order.

CLI

Rename directory contents with prefix of directory

Leave a comment

Quick snippet to rename every file within a directory to have a prefix of the directory they reside in as part of the file name. If the directory name has a space in it, replace spaces with underscores for the file name. Run from within the directory in question.

base=$(basename "$PWD"| tr ' ' '_'); for file in *; do [ -f "$file" ] && mv "$file" "${base}_$file"; done

It does the following:

  • Gets the name of the current directory, replacing spaces with underscores, and saves into the variable base
  • Iterates through everything in the directory in a for loop
  • If the item is a regular file, execute the mv command to rename the file to include the contents of the base variable as a prefix
    • It uses BASH substitution to prepend the directory name to the new file name

This was helpful when dealing with a scanning project where many files had the same filename in different directories, which confused stacking images within Immich.

CLI

Get list of offline hosts with ping, grep & awk

Leave a comment

Here is a simple bash one-liner that takes a list of hosts to check via stdin and attempts to ping a host a single time. If no response is received within 1 second, it prints that hostname and moves onto the next host. It’s designed to work with the output of another command that outputs hostnames (for example, an inventory file.) 

|awk '{print $6}'| xargs -I {} sh -c 'ping -c 1 -w 1 {} | grep -B1 100% |  head -1' | awk '{print $2}'
  • Prints the 6th column of the output (you may or may not need this depending on what program is outputting hostnames)
  • xargs takes the output from the previous command and runs the ping command against it in a subshell
    • ping -c1 to only do it once, -w1 to wait 1 second for timeout
    • grep for 100%, grab the line before it (100% in this case means packet loss)
    • head -1 only prints the first line of the ping results
  • Awk prints only the second column in the resulting ping statistics output

It takes output like this:

PING examplehost (10.13.12.12) 56(84) bytes of data.

— examplehost ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

And simply outputs this:

examplehost

but only if the ping failed. No output otherwise.

I will note that the Anthropic’s Claude Sonnet AI helped me come to this conclusion, but not directly. Its suggestions for my problem didn’t work but were enough to point me in the right direction. The grep -B1 100% | head -1 portion need to be grouped together with the ping command in a separate shell, not appended afterward.

CLI

Generate list of youtube links from song titles

2 Comments

I needed to get a list of youtube links from a list of song titles. Thanks to this reddit post I was able to get what I needed. I did have to update it to use yt which is a fork of the referenced mps-youtube package.

After installing yewtube per https://github.com/mps-youtube/yewtube#installation I was able to get what I wanted with this one-liner:

while read song; do echo $song; yt search "$song", i 1, q|grep -i link| awk -F ': ' '{ print $2 }'; done < playlist

The above command looks at a playlist which is only artist & song names, prints the song name to the console for reference, then uses yewtube to search youtube for that song name and select the first result, then grab the link and print it to the screen.

I had to double check that the correct version of the song was selected, but for the most part it did exactly what I needed!