All posts by nicholas

CLI

Increment modified date of files in a directory based on first file

Leave a comment

I had an issue in Immich where it was sorting pictures by their modified date. The modified dates are random, but the filenames are not. I wanted the album to sort by filename, and to do that I needed to get each filename to have a modified time in the same order. This was my solution (run within the directory in question) :

date=$(date -r $(ls | head -1) +%s); for file in *.jpg; do touch -m -d "@$date" $file; ((date+=1)); done

This bash one-liner does the following:

  • Sets a date variable by taking the modified date of the first file in the directory and converting it to epoch time
  • Goes through each JPG file in the directory and executes a touch command to set the date of that file to the date variable
  • Increments the date variable by 1 before processing the next file

The end result is now the order the files are in by modified date match their filename order.

CLI

Rename directory contents with prefix of directory

Leave a comment

Quick snippet to rename every file within a directory to have a prefix of the directory they reside in as part of the file name. If the directory name has a space in it, replace spaces with underscores for the file name. Run from within the directory in question.

base=$(basename "$PWD"| tr ' ' '_'); for file in *; do [ -f "$file" ] && mv "$file" "${base}_$file"; done

It does the following:

  • Gets the name of the current directory, replacing spaces with underscores, and saves into the variable base
  • Iterates through everything in the directory in a for loop
  • If the item is a regular file, execute the mv command to rename the file to include the contents of the base variable as a prefix
    • It uses BASH substitution to prepend the directory name to the new file name

This was helpful when dealing with a scanning project where many files had the same filename in different directories, which confused stacking images within Immich.

Networking

Wireguard client full tunnel no internet

Leave a comment

I tried to set up a wireguard full tunnel VPN on my Debian Bookworm server and ran into issues with internet connectivity. LAN / VPN connectivity was fine, just no internet.

My first realization was that when making changes to your config, be sure to wg-quick down & wg-quick up (if you’re using wg-quick.) Simply editing the files and reloading the service doesn’t pick up the changes.

I followed this guide to get it up, and it simply wasn’t working:

https://wiki.debian.org/WireGuard#Step_2_-Alternative_A-_Manual_Configuration

I did eventually realize I need to enable the following in /etc/sysctl.conf

net.ipv4.ip_forward = 1

Then reload settings with:

# sysctl -p

In the wireguard server config I added these iptables commands:
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

I discovered that allowed IPs on the server side are simply the IP address(es) of the wireguard clients, nothing more.

For full tunnel, set Client’s allowed IP to 0.0.0.0/0

I did all this and it still didn’t work. Then I stumbled upon https://www.ckn.io/blog/2017/11/14/wireguard-vpn-typical-setup

which mentioned to enable conntrack with iptables:

iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

When I enabled conntrack, internet connectivity worked. I decided to reboot without making the above iptables commands persistent. But it worked after reboot!

Lesson learned: try rebooting the host as a wireguard troubleshooting step, especially if all the configs look like they should be working but simply aren’t.

Here are my working configs:

Server:

[Interface]
Address = 10.10.1.1/24
SaveConfig = true
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
ListenPort = 51820
PrivateKey = <server_private_key>

[Peer]
PublicKey = <client_public_key>
AllowedIPs = 10.10.1.5/32

Client:

[Interface]
Address = 10.10.1.5/32
PrivateKey = <private key>
DNS = 10.10.1.1 10.10.1.2

[Peer]
PublicKey = <server public key>
Endpoint = mx.jeppson.org:54137
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 21

CLI

Get list of offline hosts with ping, grep & awk

Leave a comment

Here is a simple bash one-liner that takes a list of hosts to check via stdin and attempts to ping a host a single time. If no response is received within 1 second, it prints that hostname and moves onto the next host. It’s designed to work with the output of another command that outputs hostnames (for example, an inventory file.) 

|awk '{print $6}'| xargs -I {} sh -c 'ping -c 1 -w 1 {} | grep -B1 100% |  head -1' | awk '{print $2}'
  • Prints the 6th column of the output (you may or may not need this depending on what program is outputting hostnames)
  • xargs takes the output from the previous command and runs the ping command against it in a subshell
    • ping -c1 to only do it once, -w1 to wait 1 second for timeout
    • grep for 100%, grab the line before it (100% in this case means packet loss)
    • head -1 only prints the first line of the ping results
  • Awk prints only the second column in the resulting ping statistics output

It takes output like this:

PING examplehost (10.13.12.12) 56(84) bytes of data.

— examplehost ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

And simply outputs this:

examplehost

but only if the ping failed. No output otherwise.

I will note that the Anthropic’s Claude Sonnet AI helped me come to this conclusion, but not directly. Its suggestions for my problem didn’t work but were enough to point me in the right direction. The grep -B1 100% | head -1 portion need to be grouped together with the ping command in a separate shell, not appended afterward.

CLI

Generate list of youtube links from song titles

2 Comments

I needed to get a list of youtube links from a list of song titles. Thanks to this reddit post I was able to get what I needed. I did have to update it to use yt which is a fork of the referenced mps-youtube package.

After installing yewtube per https://github.com/mps-youtube/yewtube#installation I was able to get what I wanted with this one-liner:

while read song; do echo $song; yt search "$song", i 1, q|grep -i link| awk -F ': ' '{ print $2 }'; done < playlist

The above command looks at a playlist which is only artist & song names, prints the song name to the console for reference, then uses yewtube to search youtube for that song name and select the first result, then grab the link and print it to the screen.

I had to double check that the correct version of the song was selected, but for the most part it did exactly what I needed!

Automation, Hardware

Report proper RATGDO door state in home assistant

Leave a comment

I have an old dry contact garage door opener. I’ve wired it to a nifty ratgdo device to control it with Home Assistant. I ran into an issue with wiring reed switches to report the garage door state (open / closed.) The wires carrying the data would pick up RFI that made my remotes stop working. I never did find a way around this wired issue, so I went wireless and installed a Zigbee door sensor.

I struggled to get Home Assistant to report the state of the door as reported by the sensor. After much reading of documentation I finally got it working! Here is my configuration.yml:

cover:

  – platform: template

    covers:

      east_garage_door:

        device_class: garage

        friendly_name: “East Garage Door”

        unique_id: “east_garage_door”

        value_template: “{{ is_state(‘binary_sensor.east_garage_door_opening’, ‘on’) }}”

        open_cover:

          – action: button.press

            target:

              entity_id: button.east_garage_door_toggle_door

        close_cover:

          – action: button.press

            target:

              entity_id: button.east_garage_door_toggle_door

        stop_cover:

          action: button.press

          target:

            entity_id: button.east_garage_door_toggle_door

CLI, Hardware, OS, Virtualization

Add NVIDIA GPU to LXC container

Leave a comment

I followed this guide to get NVIDIA drivers working on my Proxmox machine. However when I tried to get them working in my container I couldn’t see how to get nvidia-smi installed. Thankfully this blog had what I needed.

The step I missed was copying & installing the NVIDIA drivers into the container with this flag:

--no-kernel-module

That got me one step closer but I could not spin up open-webui in a container. I kept getting the error 

Error response from daemon: could not select device driver "nvidia" with capabilities: [[gpu]]

The fix was to install the NVDIA Container Toolkit:

Configure the production repository:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Update the packages list from the repository:

sudo apt-get update

Install the NVIDIA Container Toolkit packages:

sudo apt-get install -y nvidia-container-toolkit

An additional hurtle I encountered was this error:

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: mount error: failed to add device rules: unable to find any existing device filters attached to the cgroup: bpf_prog_query(BPF_CGROUP_DEVICE) failed: operation not permitted: unknown

I found here that the fix is to change a line in /etc/nvidia-container-runtime/config.toml. Uncomment and change no-cgroups to true.

no-cgroups = true

Success.

Not working after reboot

I had a working config until I rebooted the host. It turns out that two services need to run on the host:

nvidia-persistenced
nvidia-smi

Configured cron tab to run these on reboot:

/etc/cron.d/nvidia:
@reboot root /usr/bin/nvidia-smi
@reboot root /usr/bin/nvidia-persistenced

Update 2025-05-06

I encountered an error when trying to set up alltalk tts:


nvidia-container-cli: mount error: stat failed: /dev/nvidia-modeset: no such file or directory: unknown

It turns out I needed to expose /dev/nvidia-modeset to the container as well. Thanks to this reddit post for the answer. The complete container passthrough config is now this:

lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 243:* rwm
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file

CLI

Recursively find files with the same filename

Leave a comment

I needed a way to find files with the same filename, but were not identical files. Thankfully Reddit had the solution I was looking for: a combination of find, sort, and while loop with if statements.

https://www.reddit.com/r/bash/comments/fjsr8v/recursively_find_files_with_same_name_under_a/

find . -type f -printf '%f/%p\0' | { sort -z -t/ -k1; printf '\0';} |
while IFS=/ read -r -d '' name file; do
    if [[ "$name" = "$oldname" ]]; then
        repeated+=("$file")  # duplicate file
        continue
    fi
    if (( ${#repeated[@]} > 1)); then
        printf '%s\n' "$oldname" "${repeated[@]}" ''
        # do something with list "${repeated[@]}"
    fi
    repeated=("$file")
    oldname=$name
done
Web

Load balancing behind Nginx Proxy Manager

Leave a comment

Nginx proxy manager is a really convenient UI wrapped around nginx. It covers the most common use cases very well. If you have more advanced needs, then it requires some custom configuration. In my case, I wanted to load balance my Proxmox servers. This is how you do that, as per https://nginxproxymanager.com/advanced-config/#custom-nginx-configurations and https://www.reddit.com/r/selfhosted/comments/1fp5mxz/nginx_proxy_manager_fails_when_adding_load/

  1. Make directories & create data/nginx/custom/root_top.conf
  2. Populate with your upstream server list
    upstream proxmox-backend {
    server first_server.fqdn:8006;
    server second_server.fqdn:8006;
    server nth_server.fqdn:8006;
    }
  3. Set proxy host for http://backend and edit proxy host to have the following in the Advanced tab:
    location / {
    proxy_pass https://proxmox-backend;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    }
    4.Profit

In my case it is very simple load balancing with no persistent cookies/session (not needed for my application.) But all those options exist for nginx and it’s just a matter of adding them in custom configuration for NPM.

CLI, Web

Configure Proxmox Mail Gateway to use AnyMX relay

1 Comment

I needed to configure Proxmox Mail Gateway to use an authenticated SMTP relay for outgoing mail. There is no way to add a username and password in the PMG GUI; however, you can do it in the command line and it follows standard postfix syntax. Thanks to this post for helping me get it set up: https://forum.proxmox.com/threads/relay-username-and-password.129586

To get it to work you have to drop to the CLI and configure your username and password. Then copy the template file over and make your changes there, as editing postfix directly gets overwritten with subsequent GUI changes.

mkdir /etc/pmg/templates/

cp /var/lib/pmg/templates/main.cf.in /etc/pmg/templates/main.cf.in

create /etc/postfix/smtp_auth and populate it:

relay.host.tld   username:password

Append the following to the /etc/pmg/templates/main.cf.in template:

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/smtp_auth
smtp_sasl_security_options = noanonymous

Change permissions to smtp_auth file, run postmap to generate the db with the password, and run pmgconfig to refresh the configuration.

chmod 640 /etc/postfix/smtp_auth
postmap /etc/postfix/smtp_auth
pmgconfig sync --restart 1

This worked for me.