Get a summary of disk usage from select files with find, sed, du, and xargs

I wanted a quick way in the command line to get the disk usage of a bunch of zip files I downloaded in the previous day. I also wanted them sorted by filename and to have quotes surround each filename. I learned from this stackexchange post that du -ch is the command I want to accomplish this. Here is my final command. It works! Note: I ran this on a mac, so I had to use gsed because the version of sed that ships with mac is rather crippled. On linux the command would simply be sed instead of gsed

find . -name "*.zip" -mtime -1|sort -h|sed 's/.\//"/g'|sed 's/.zip/.zip"/g'|gsed -z 's/\n/ /g'|xargs du -ch

The output looks like this (snippet – not the full output):

753M V-A – Mixed by Mahiane – OXYCANTA.zip
912M V-A – Mixed by Nova – ALBEDO.zip
816M V-A – Selected by Fishimself – AMBROSIA (24bits).zip
977M Various Artists – FAHRENHEIT PROJECT – Part 1.zip
992M Various Artists – FAHRENHEIT PROJECT – Part 2.zip
848M Various Artists – FAHRENHEIT PROJECT – Part 3.zip
849M Various Artists – FAHRENHEIT PROJECT – Part 4.zip
817M Various Artists – FAHRENHEIT PROJECT – Part 5.zip
897M Various Artists – FAHRENHEIT PROJECT – Part 6.zip
897M Various Artists – FAHRENHEIT PROJECT – Part 7.zip
737M Various Artists – ISOLATED (24bit).zip
817M Various Artists – OPIA (24bit).zip
55G total

For the curious, I had purchased the Ultimae Digital Collection. Great stuff.

Site to Site VPN with Tailscale subnet router

My manual wireguard site to site solution worked but had latency issues. I wanted a more streamlined way to get my site to site VPN working properly. I decided to finally try out tailscale but didn’t want to rely on their servers, so I spun up headscale and hosted the control server myself.

My sites have disparate routers which don’t lend to installing the tailscale client, so I opted to spin up dedicated subnet router nodes and then tell the firewalls at each site to forward the routes for the other sites’ subnets to their local subnet router.

The documentation is quite good and it didn’t take long for me to get a working solution.

Configuration

  • Install headscale
    • Configure URL, DNS, ACL
    • Allow all: {}
    • sudo docker exec headscale <command>
  • Set up subnet routers
    • Advertise routes: sudo tailscale set --advertise-routes=192.0.2.0/24,198.51.100.0/24
  • Advertise exit node, specify login server, set hostname, accept routes
    • sudo tailscale up --hostname <HOSTNAME> --login-server=<HEADSCALE_URL> --accept-routes --advertise-exit-node
  • Accept routing on the control server
    • sudo docker exec headscale headscale nodes list-route
    • sudo docker exec headscale headscale nodes approve-routes -i <ID> -r <SUBNET>
  • Add tailscale interface as trusted interface
    • sudo firewall-cmd --zone=trusted --add-interface=tailscale0 --permanent
  • Configure docker DNS to use tailscale’s magic DNS
    • /etc/docker/daemon.json:
      {"dns": ["100.100.100.100","1.1.1.1","9.9.9.9"]}

Troubleshooting

CONFIG_TUN error

is CONFIG_TUN enabled in your kernel? modprobe tun failed with: modprobe: FATAL: Module tun not found in directory /lib/modules/6.8.8-4-pve

Solution found here: https://diegocarrasco.com/install-tailscale-proxmox-lxc-container-almalinux-9

You need to edit the conf file for your LXC and allow/mount /dev/net/tun to your container:

vi /etc/pve/lxc/<LXC_NUMBER>.conf
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

Port Forward from Internet to Wireguard interface

I needed to give my CGNAT-backed home internet a way to have a public IP address. My first solution was to use wireguard directly, and forward ports as needed. I came across this article that helped me do it. The key was to enable packed masquerading so the return path could be completed. Example wireguard server config:

# packet forwarding
PreUp = sysctl -w net.ipv4.ip_forward=1

# port forwarding
PreUp = iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 2000 -j DNAT --to-destination 10.0.0.1:8080
PostDown = iptables -t nat -D PREROUTING -i eth0 -p tcp --dport 2000 -j DNAT --to-destination 10.0.0.1:8080

# packet masquerading
PreUp = iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
PostDown = iptables -t nat -D POSTROUTING -o wg0 -j MASQUERADE

Example wireguard client config:

PreUp = iptables -t nat -A POSTROUTING -o wg0
PostUp = iptables -A FORWARD -i %i -j ACCEPT
PostDown = iptables -D FORWARD -i %i -j ACCEPT
PostDown = iptables -t nat -D POSTROUTING -o wg0

Make sure you have correct allowedIPs configured on client and server. This does work, but it shows the source IP as being the VPN destination. If you value seeing what true external source IPs are, then this solution is not for you (eg seeing external IPs accessing a webserver.)

DNS resolution inside docker containers

I had an issue where docker containers weren’t resolving DNS properly over this VPN tunnel. I found this site that explained I needed to update my docker daemon.json to explicitly specify which DNS servers to use, then restart docker:

{
  "dns": ["172.17.0.1","10.10.10.1"]
}

Troubleshoot blackbox exporter errors

I had a frustrating issue where prometheus blackbox exporter wasn’t able to check one of my websites and I couldn’t figure out why. I finally found this site which explained you can append &debug=true to the end of your probe in a URL string talking directly to blackbox. For example:

http://prometheus:9115/probe?module=http_2xx&target=http://customsite.com:8096/web/&debug=true

That finally got me to see what the problem was. It was resolving to an IPv6 address, but I didn’t have my IPv6 stack properly configured. I then discovered this site which led me to the solution: prefer IPv4. I appended this to my http_2xx config:

http_2xx:
  prober: http
  http:
    preferred_ip_protocol: ip4
...

That fixed the issue!