Category Archives: CLI

CLI

Get a summary of disk usage from select files with find, sed, du, and xargs

Leave a comment

I wanted a quick way in the command line to get the disk usage of a bunch of zip files I downloaded in the previous day. I also wanted them sorted by filename and to have quotes surround each filename. I learned from this stackexchange post that du -ch is the command I want to accomplish this. Here is my final command. It works! Note: I ran this on a mac, so I had to use gsed because the version of sed that ships with mac is rather crippled. On linux the command would simply be sed instead of gsed

find . -name "*.zip" -mtime -1|sort -h|sed 's/.\//"/g'|sed 's/.zip/.zip"/g'|gsed -z 's/\n/ /g'|xargs du -ch

The output looks like this (snippet – not the full output):

753M V-A – Mixed by Mahiane – OXYCANTA.zip
912M V-A – Mixed by Nova – ALBEDO.zip
816M V-A – Selected by Fishimself – AMBROSIA (24bits).zip
977M Various Artists – FAHRENHEIT PROJECT – Part 1.zip
992M Various Artists – FAHRENHEIT PROJECT – Part 2.zip
848M Various Artists – FAHRENHEIT PROJECT – Part 3.zip
849M Various Artists – FAHRENHEIT PROJECT – Part 4.zip
817M Various Artists – FAHRENHEIT PROJECT – Part 5.zip
897M Various Artists – FAHRENHEIT PROJECT – Part 6.zip
897M Various Artists – FAHRENHEIT PROJECT – Part 7.zip
737M Various Artists – ISOLATED (24bit).zip
817M Various Artists – OPIA (24bit).zip
55G total

For the curious, I had purchased the Ultimae Digital Collection. Great stuff.

CLI, Networking

Port Forward from Internet to Wireguard interface

1 Comment

I needed to give my CGNAT-backed home internet a way to have a public IP address. My first solution was to use wireguard directly, and forward ports as needed. I came across this article that helped me do it. The key was to enable packed masquerading so the return path could be completed. Example wireguard server config:

# packet forwarding
PreUp = sysctl -w net.ipv4.ip_forward=1

# port forwarding
PreUp = iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 2000 -j DNAT --to-destination 10.0.0.1:8080
PostDown = iptables -t nat -D PREROUTING -i eth0 -p tcp --dport 2000 -j DNAT --to-destination 10.0.0.1:8080

# packet masquerading
PreUp = iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
PostDown = iptables -t nat -D POSTROUTING -o wg0 -j MASQUERADE

Example wireguard client config:

PreUp = iptables -t nat -A POSTROUTING -o wg0
PostUp = iptables -A FORWARD -i %i -j ACCEPT
PostDown = iptables -D FORWARD -i %i -j ACCEPT
PostDown = iptables -t nat -D POSTROUTING -o wg0

Make sure you have correct allowedIPs configured on client and server. This does work, but it shows the source IP as being the VPN destination. If you value seeing what true external source IPs are, then this solution is not for you (eg seeing external IPs accessing a webserver.)

DNS resolution inside docker containers

I had an issue where docker containers weren’t resolving DNS properly over this VPN tunnel. I found this site that explained I needed to update my docker daemon.json to explicitly specify which DNS servers to use, then restart docker:

{
  "dns": ["172.17.0.1","10.10.10.1"]
}
CLI

Increment modified date of files in a directory based on first file

Leave a comment

I had an issue in Immich where it was sorting pictures by their modified date. The modified dates are random, but the filenames are not. I wanted the album to sort by filename, and to do that I needed to get each filename to have a modified time in the same order. This was my solution (run within the directory in question) :

date=$(date -r $(ls | head -1) +%s); for file in *.jpg; do touch -m -d "@$date" $file; ((date+=1)); done

This bash one-liner does the following:

  • Sets a date variable by taking the modified date of the first file in the directory and converting it to epoch time
  • Goes through each JPG file in the directory and executes a touch command to set the date of that file to the date variable
  • Increments the date variable by 1 before processing the next file

The end result is now the order the files are in by modified date match their filename order.

CLI

Rename directory contents with prefix of directory

Leave a comment

Quick snippet to rename every file within a directory to have a prefix of the directory they reside in as part of the file name. If the directory name has a space in it, replace spaces with underscores for the file name. Run from within the directory in question.

base=$(basename "$PWD"| tr ' ' '_'); for file in *; do [ -f "$file" ] && mv "$file" "${base}_$file"; done

It does the following:

  • Gets the name of the current directory, replacing spaces with underscores, and saves into the variable base
  • Iterates through everything in the directory in a for loop
  • If the item is a regular file, execute the mv command to rename the file to include the contents of the base variable as a prefix
    • It uses BASH substitution to prepend the directory name to the new file name

This was helpful when dealing with a scanning project where many files had the same filename in different directories, which confused stacking images within Immich.

CLI

Get list of offline hosts with ping, grep & awk

Leave a comment

Here is a simple bash one-liner that takes a list of hosts to check via stdin and attempts to ping a host a single time. If no response is received within 1 second, it prints that hostname and moves onto the next host. It’s designed to work with the output of another command that outputs hostnames (for example, an inventory file.) 

|awk '{print $6}'| xargs -I {} sh -c 'ping -c 1 -w 1 {} | grep -B1 100% |  head -1' | awk '{print $2}'
  • Prints the 6th column of the output (you may or may not need this depending on what program is outputting hostnames)
  • xargs takes the output from the previous command and runs the ping command against it in a subshell
    • ping -c1 to only do it once, -w1 to wait 1 second for timeout
    • grep for 100%, grab the line before it (100% in this case means packet loss)
    • head -1 only prints the first line of the ping results
  • Awk prints only the second column in the resulting ping statistics output

It takes output like this:

PING examplehost (10.13.12.12) 56(84) bytes of data.

— examplehost ping statistics —
1 packets transmitted, 0 received, 100% packet loss, time 0ms

And simply outputs this:

examplehost

but only if the ping failed. No output otherwise.

I will note that the Anthropic’s Claude Sonnet AI helped me come to this conclusion, but not directly. Its suggestions for my problem didn’t work but were enough to point me in the right direction. The grep -B1 100% | head -1 portion need to be grouped together with the ping command in a separate shell, not appended afterward.

CLI

Generate list of youtube links from song titles

2 Comments

I needed to get a list of youtube links from a list of song titles. Thanks to this reddit post I was able to get what I needed. I did have to update it to use yt which is a fork of the referenced mps-youtube package.

After installing yewtube per https://github.com/mps-youtube/yewtube#installation I was able to get what I wanted with this one-liner:

while read song; do echo $song; yt search "$song", i 1, q|grep -i link| awk -F ': ' '{ print $2 }'; done < playlist

The above command looks at a playlist which is only artist & song names, prints the song name to the console for reference, then uses yewtube to search youtube for that song name and select the first result, then grab the link and print it to the screen.

I had to double check that the correct version of the song was selected, but for the most part it did exactly what I needed!

CLI, Hardware, OS, Virtualization

Add NVIDIA GPU to LXC container

Leave a comment

I followed this guide to get NVIDIA drivers working on my Proxmox machine. However when I tried to get them working in my container I couldn’t see how to get nvidia-smi installed. Thankfully this blog had what I needed.

The step I missed was copying & installing the NVIDIA drivers into the container with this flag:

--no-kernel-module

That got me one step closer but I could not spin up open-webui in a container. I kept getting the error 

Error response from daemon: could not select device driver "nvidia" with capabilities: [[gpu]]

The fix was to install the NVDIA Container Toolkit:

Configure the production repository:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Update the packages list from the repository:

sudo apt-get update

Install the NVIDIA Container Toolkit packages:

sudo apt-get install -y nvidia-container-toolkit

An additional hurtle I encountered was this error:

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: mount error: failed to add device rules: unable to find any existing device filters attached to the cgroup: bpf_prog_query(BPF_CGROUP_DEVICE) failed: operation not permitted: unknown

I found here that the fix is to change a line in /etc/nvidia-container-runtime/config.toml. Uncomment and change no-cgroups to true.

no-cgroups = true

Success.

Not working after reboot

I had a working config until I rebooted the host. It turns out that two services need to run on the host:

nvidia-persistenced
nvidia-smi

Configured cron tab to run these on reboot:

/etc/cron.d/nvidia:
@reboot root /usr/bin/nvidia-smi
@reboot root /usr/bin/nvidia-persistenced

Update 2025-05-06

I encountered an error when trying to set up alltalk tts:


nvidia-container-cli: mount error: stat failed: /dev/nvidia-modeset: no such file or directory: unknown

It turns out I needed to expose /dev/nvidia-modeset to the container as well. Thanks to this reddit post for the answer. The complete container passthrough config is now this:

lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 243:* rwm
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file

CLI

Recursively find files with the same filename

Leave a comment

I needed a way to find files with the same filename, but were not identical files. Thankfully Reddit had the solution I was looking for: a combination of find, sort, and while loop with if statements.

https://www.reddit.com/r/bash/comments/fjsr8v/recursively_find_files_with_same_name_under_a/

find . -type f -printf '%f/%p\0' | { sort -z -t/ -k1; printf '\0';} |
while IFS=/ read -r -d '' name file; do
    if [[ "$name" = "$oldname" ]]; then
        repeated+=("$file")  # duplicate file
        continue
    fi
    if (( ${#repeated[@]} > 1)); then
        printf '%s\n' "$oldname" "${repeated[@]}" ''
        # do something with list "${repeated[@]}"
    fi
    repeated=("$file")
    oldname=$name
done
CLI, Web

Configure Proxmox Mail Gateway to use AnyMX relay

1 Comment

I needed to configure Proxmox Mail Gateway to use an authenticated SMTP relay for outgoing mail. There is no way to add a username and password in the PMG GUI; however, you can do it in the command line and it follows standard postfix syntax. Thanks to this post for helping me get it set up: https://forum.proxmox.com/threads/relay-username-and-password.129586

To get it to work you have to drop to the CLI and configure your username and password. Then copy the template file over and make your changes there, as editing postfix directly gets overwritten with subsequent GUI changes.

mkdir /etc/pmg/templates/

cp /var/lib/pmg/templates/main.cf.in /etc/pmg/templates/main.cf.in

create /etc/postfix/smtp_auth and populate it:

relay.host.tld   username:password

Append the following to the /etc/pmg/templates/main.cf.in template:

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/smtp_auth
smtp_sasl_security_options = noanonymous

Change permissions to smtp_auth file, run postmap to generate the db with the password, and run pmgconfig to refresh the configuration.

chmod 640 /etc/postfix/smtp_auth
postmap /etc/postfix/smtp_auth
pmgconfig sync --restart 1

This worked for me.

CLI, OS

Change ceph network

Leave a comment

My notes on changing which network your Proxmox CEPH cluster lives in. In my case I wanted to switch from a 10 gig network to a 40gig network in a different subnet. Source: https://forum.proxmox.com/threads/ceph-changing-public-network.119116

  1. Change network configuration in “ceph.conf”
    • Be sure to edit both cluster network and public network
  2. Destroy and recreate monitors (one by one);
  3. Destroy and recreate managers (one by one, leaving the active one for last);
  4. Destroy and recreate metadata servers (one by one, leaving the active one for last;
  5. Restart OSDs (one by one – or more, depending how many OSDs you have in the cluster – so you avoid restarting the hosts);

Get CEPH running on new Proxmox node

pveceph install –repository no-subscription

Move OSDs to new host

Source: https://forum.proxmox.com/threads/move-osd-to-another-node.33965/page-2

Follow a similar procedure above of downing each OSD one by one on the old host. Remove the drives and place them in the new host. Then run the following:

pvscan
ceph-volume lvm activate --all

Troubleshooting

Unable to remove monitor with unknown status

https://forum.proxmox.com/threads/ceph-cant-remove-monitor-with-unknown-status.63613

rm -r /var/lib/ceph/mon/ceph-pve2/

Remove failed host

I had to edit /etc/pve/ceph.conf manually, remove host when it failed. It wouldn’t work in the Proxmox GUI.