All posts by nicholas

Powershell equivalent of “find -exec”

I recently found myself on a Windows 10 system needing to do the equivalent of “find . -name *.mdi -exec mdiconvert -source {} -log log.txt \;” I knew what to do instantly on a Unix system, not so on a Windows system

I finally figured it out and am now writing it down because I know I’ll forget! Thanks to these several sites for pointing me in the right direction.

  • Get-ChildItem is the find equivalent
    • -Filter is the -name equivalent
    • -Recurse must be specified otherwise it only looks in the one directory
  • % is an alias for “ForEach-Object
  • Put the command you want run in brackets {}
  • Put an ampersand in front of the command you wish to run so you can properly pass arguments containing dashes
  • $_.FullName turns the powershell object into a text string (which my command required.) FullName is the full path of the item found with Get-ChildItem
    • $_ is the rough equivalent of find’s {} (the item that was found)

The command I ended up using is below (find any .mdi files and use Microsoft’s mdi2tif utility to convert the result to .tif files)

Get-ChildItem "C:\Users" -Recurse -Filter *.mdi | % { & 'C:\Program
Files (x86)\modiconv\MDI2TIF.EXE' -source $_.FullName -log log.txt }

Make FreeDOS boot ISO to flash BIOS

I needed to flash the BIOS of one of my old server motherboards and to my dismay found the only way to do so was via DOS boot image. It was not straightforward so I thought I’d write it down. Thanks to pingtool & tummy.com for the information I needed to pull it off.

First, obtain a copy of FreeDOS ISO and extract it to a directory

  • mount -o loop <freedosISO.iso> <mount directory>
  • rsync -aP <mount directory> <directory you want files to copy to>

Next, copy the necessary flash utilities and firmware files to that same directory as above.

Lastly, use genisoimage to create a new ISO image based on the above folder. Modify -o output to wherever you want the ISO to go.

sudo apt install genisoimage
cd <folder you copied your files to>
mkisofs -o /tmp/freedos_biosupdate.iso -q -l -N \
   -boot-info-table -iso-level 4 -no-emul-boot \
   -b isolinux/isolinux.bin \
   -publisher "FreeDOS - www.freedos.org" \
   -A "FreeDOS beta9 Distribution" -V FDOS_BETA9 -v .

From here you can take the ISO and mount / burn it as needed. It will boot into FreeDOS. Tell it to go to a shell and away you go.

FreeNAS ZFS tuning for SSDs

I wanted to optimize my all SSD storage array on my FreeNAS server but I had a hard time finding information in one place. After a lot of digging I pulled things from several places. This is what I came up with. It boiled down to two main settings

  • ashift
  • recordsize

Checking ashift on existing pools

zdb -U /data/zfs/zpool.cache | grep ashift

I read here a recommended setting of ashift=13, recordsize=8k for VM workloads on SSDs.

How to change recordsize:

This is easily done in the GUI or command line and can be changed on the fly.

zfs set recordize <value> <volume>

How to change ashift:

Backup your data and destroy the pool.

Modify the setting dictating minimum ashift setting as outlined here

sysctl vfs.zfs.min_auto_ashift=13

Re-create the pool.

Note: this lustre wiki actually recommends ashift 12 instead of 13.

UPDATE 2/26/20: You can also set the ashift at pool creation time as dictated here

zpool create POOL_NAME -o ashift=12 ...

Additional reading

http://open-zfs.org/wiki/Performance_tuning#Alignment_shift
https://www.reddit.com/r/zfs/comments/7pfutp/zfs_pool_planning_for_vm_storage/

Free up RAM after Proxmox live migration

I ran into an issue where after migrating a bunch of VMs off of one of  my hosts, the remaining VMs on it refused to turn on. Every time I tried the command would hang for a while and eventually error out with this message

TASK ERROR: start failed: command '/usr/bin/kvm -id <truncated>... ' failed: got timeout

I suspected this might be due to RAM use and sure enough the usage was too high for a system that didn’t have any VMs running on it.  I found here that I could run a command to flush the cache:

echo 3 > /proc/sys/vm/drop_caches

That caused the RAM usage to go down but the symptom of the VM not starting remained. I then saw the KSM sharing still had some memory in it. I decided to restart the KSM sharing service:

sudo systemctl restart ksmtuned

After running that the VM started!

Supermicro fans constantly spinning to 100% fix

My fancy new Supermicro-powered AMD Epyc 7 series server is the bee’s knees. When I first set up I had an annoying problem though – the fans would spin to 100% and back down to normal speeds constantly. Logs kept spamming the same thing over and over:

SENSOR_NUMBER: 45
SENSOR_TYPE: Fan
SENSOR_NAME: FAN5            
EVENT_DESCRIPTION: Lower Critical going low
EVENT_DIRECTION: Assertion
EVENT SEVERITY:"information"
SENSOR_NUMBER: 45
SENSOR_TYPE: Fan
SENSOR_NAME: FAN5            
EVENT_DESCRIPTION: Lower Critical going low
EVENT_DIRECTION: De-assertion
EVENT SEVERITY:"information"

It was doing this for all 3 fans I had plugged in there. I finally came across this post which explained what the problem was. The fans I had installed defaulted to a low RPM mode, too low for the motherboard’s liking. The BMC would detect the low RPM and assume something was wrong and bring all fans to 100% in order to rescue the system. After doing this it would see the RPM go back to normal range and turn off the “emergency fan mode” only to turn back on when the RPMs of my fans went below the threshold.

The fix, thankfully, is pretty simple provided you have ipmitools installed. One simply has to use the tool to change the fan thresholds. For my Debian-based Proxmox install I did the following to quiet this beast:

apt install ipmitool
ipmitool sensor thresh FAN1 lower 300 300 400
ipmitool sensor thresh FAN2 lower 300 300 400
ipmitool sensor thresh FAN5 lower 300 300 400
#you'll want to modify the fans to correspond with your own server.

I ran the above commands to turn my 3 fans back to a sane speed. I got caught up for a while because I only ran this command on 2 of my 3 fans at first. The deafening noise continued. This is because if any fan in the system goes below, all fans spin up. Once I changed that third fan’s threshold all was gravy. My ears were ringing for a while, but they’re fine now.

Track and log unclean shutdowns in CentOS 7

I needed to find a way to track if my CentOS 7 systems reboot unexpectedly. I was surprised that this isn’t something that the OS does by default. I found this article from RedHat that outlines that you basically have to write a couple of systemd scripts yourself if you want this functionality. So, I did.

I ended up with three separate systemd services that accomplish what I want:

  • set_graceful_shutdown: Runs just before shutdown. Creates a file /root/grateful_shutdown
  • log_ungraceful_shutdown: Runs on startup. Checks to see if /root/grateful_shutdown is missing and logs this fact to a file (/var/log/shutdown.log) if it is.
  • reset_shutdown_flag: Runs after log_ungraceful_shutdown. It checks for the presence of that file, and if it exists, removes it.

I placed these three files into /etc/systemd/system and then ran systemctl daemon-reload & systemctl enable for each one.

set_graceful_shutdown.service

[Unit]
Description=Set flag for graceful shutdown
DefaultDependencies=no
RefuseManualStart=true
Before=shutdown.target

[Service]
Type=oneshot
ExecStart=/bin/touch /root/graceful_shutdown

[Install]
WantedBy=shutdown.target

log_ungraceful_shutdown.service

[Unit]
Description=Log ungraceful shutdown
ConditionPathExists=!/root/graceful_shutdown
RefuseManualStart=true
RefuseManualStop=true
Before=reset_shutdown_flag.service

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/sh -c "echo $$(date): Improper shutdown detected >> /var/log/shutdown.log"

[Install]
WantedBy=multi-user.target

reset_shutdown_flag.service

[Unit]
Description=Check if previous system shutdown was graceful
ConditionPathExists=/root/graceful_shutdown
RefuseManualStart=true
RefuseManualStop=true

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/rm /root/graceful_shutdown

[Install]
WantedBy=multi-user.target

It feels like a kludge but it works pretty well. The result is I get an entry in a log file if the system wasn’t shut down properly.

Setup remote git repository with SSH & GIT

I wanted to set up a simple git repository to synchronize my bash scripts between a couple hosts, no fancy github or gitlab software required. These are my notes on how I got it working. Thanks to this site for the information.

On the remote host (server)

mkdir GIT_PROJECT_DIR.git
cd GIT_PROJECT_DIR.git
git init --bare

On the local hosts (client)

Create a git repository and add files to it:

cd GIT_FOLDER
git init
git add *
git commit -m "Initial commit"
git remote add origin USER@REMOTE_HOST:GIT_PROJECT_DIR.git
git push origin master
git branch --set-upstream-to=origin/master

Clone AD Group Memberships with Powershell

I needed to do windoze administration today.  I dug my way into a hole and finally found my way out. Thanks to answers on Technet for the information I found a way to clone AD group membership from one group to another via powershell.

The command I settled on was the following (after creating the new group first)

Get-ADGroup -Identity "name_of_existing_group" -Properties MemberOf | foreach {$_.MemberOf} | foreach {add-ADGroupMember -Identity $_ -members "new_group_to_copy_memberOf_to" }

It grabs everything the group is a member of and transforms the output to a string array. It then takes that array and uses each item of it to add your new group as a member to everything the old group is a member of.

Modify network buffers in Linux using sysctl

After installing Netdata I kept getting alerts that packets were being dropped. After some research I read you can alleviate this by increasing your system’s networking buffer. You can accomplish this with sysctl.

If you run sysctl manually it will not survive a reboot. If you want persistence you simply add the sysctl command line to /etc/sysctl.conf. Thanks to Brennen Smith on serverfault for the info on my particular problem with UDP drops:

#as root:
echo "net.core.rmem_default=8388608" >> /etc/sysctl.conf
echo "net.core.rmem_max=16777216" >> /etc/sysctl.conf
sysctl -p

The above solved my udp problem but you can modify it to change any sysctl things you want to quickly configure and persist.

Rewrite HTTP host request with Varnish

I had a stubborn WordPress plugin that somehow was remembering the URL of the site it was installed on. It became a problem when I changed the site URL. Despite changing the URL everywhere I could think of, this particular plugin was calling CSS files for the URL of the original site. I did a search and replace in the site database and searched all files for any reference to that site but couldn’t find anything. I never did find the culprit. My workaround was to use varnish to rewrite the request before it hit the browser.

Thanks to this answer by Jorge Nerin on Stack Overflow, I found my answer on how to do this.

backend www {
  .host = "www.example.com";
  .port = "http";
}

sub vcl_recv {
  if (req.http.host ~ "(?i)^(www.)?example.com$") {
    set req.backend_hint = www;
  }
}

In my case I had a default backend (no other backends configured) so my varnish config was simply adding these line in sub vcs_recv (varnish 4 syntax)

if (req.http.host ~ "(?i)^(www.)?old.host.name$") {
     set req.backend_hint = default;
}

That did the trick!