Tag Archives: linux

CLI, Hardware, OS

Make FreeDOS boot ISO to flash BIOS

3 Comments

I needed to flash the BIOS of one of my old server motherboards and to my dismay found the only way to do so was via DOS boot image. It was not straightforward so I thought I’d write it down. Thanks to pingtool & tummy.com for the information I needed to pull it off.

First, obtain a copy of FreeDOS ISO and extract it to a directory

  • mount -o loop <freedosISO.iso> <mount directory>
  • rsync -aP <mount directory> <directory you want files to copy to>

Next, copy the necessary flash utilities and firmware files to that same directory as above.

Lastly, use genisoimage to create a new ISO image based on the above folder. Modify -o output to wherever you want the ISO to go.

sudo apt install genisoimage
cd <folder you copied your files to>
mkisofs -o /tmp/freedos_biosupdate.iso -q -l -N \
   -boot-info-table -iso-level 4 -no-emul-boot \
   -b isolinux/isolinux.bin \
   -publisher "FreeDOS - www.freedos.org" \
   -A "FreeDOS beta9 Distribution" -V FDOS_BETA9 -v .

From here you can take the ISO and mount / burn it as needed. It will boot into FreeDOS. Tell it to go to a shell and away you go.

CLI, Hardware

Supermicro fans constantly spinning to 100% fix

1 Comment

My fancy new Supermicro-powered AMD Epyc 7 series server is the bee’s knees. When I first set up I had an annoying problem though – the fans would spin to 100% and back down to normal speeds constantly. Logs kept spamming the same thing over and over:

SENSOR_NUMBER: 45
SENSOR_TYPE: Fan
SENSOR_NAME: FAN5            
EVENT_DESCRIPTION: Lower Critical going low
EVENT_DIRECTION: Assertion
EVENT SEVERITY:"information"
SENSOR_NUMBER: 45
SENSOR_TYPE: Fan
SENSOR_NAME: FAN5            
EVENT_DESCRIPTION: Lower Critical going low
EVENT_DIRECTION: De-assertion
EVENT SEVERITY:"information"

It was doing this for all 3 fans I had plugged in there. I finally came across this post which explained what the problem was. The fans I had installed defaulted to a low RPM mode, too low for the motherboard’s liking. The BMC would detect the low RPM and assume something was wrong and bring all fans to 100% in order to rescue the system. After doing this it would see the RPM go back to normal range and turn off the “emergency fan mode” only to turn back on when the RPMs of my fans went below the threshold.

The fix, thankfully, is pretty simple provided you have ipmitools installed. One simply has to use the tool to change the fan thresholds. For my Debian-based Proxmox install I did the following to quiet this beast:

apt install ipmitool
ipmitool sensor thresh FAN1 lower 300 300 400
ipmitool sensor thresh FAN2 lower 300 300 400
ipmitool sensor thresh FAN5 lower 300 300 400
#you'll want to modify the fans to correspond with your own server.

I ran the above commands to turn my 3 fans back to a sane speed. I got caught up for a while because I only ran this command on 2 of my 3 fans at first. The deafening noise continued. This is because if any fan in the system goes below, all fans spin up. Once I changed that third fan’s threshold all was gravy. My ears were ringing for a while, but they’re fine now.

OS

Track and log unclean shutdowns in CentOS 7

Leave a comment

I needed to find a way to track if my CentOS 7 systems reboot unexpectedly. I was surprised that this isn’t something that the OS does by default. I found this article from RedHat that outlines that you basically have to write a couple of systemd scripts yourself if you want this functionality. So, I did.

I ended up with three separate systemd services that accomplish what I want:

  • set_graceful_shutdown: Runs just before shutdown. Creates a file /root/grateful_shutdown
  • log_ungraceful_shutdown: Runs on startup. Checks to see if /root/grateful_shutdown is missing and logs this fact to a file (/var/log/shutdown.log) if it is.
  • reset_shutdown_flag: Runs after log_ungraceful_shutdown. It checks for the presence of that file, and if it exists, removes it.

I placed these three files into /etc/systemd/system and then ran systemctl daemon-reload & systemctl enable for each one.

set_graceful_shutdown.service

[Unit]
Description=Set flag for graceful shutdown
DefaultDependencies=no
RefuseManualStart=true
Before=shutdown.target

[Service]
Type=oneshot
ExecStart=/bin/touch /root/graceful_shutdown

[Install]
WantedBy=shutdown.target

log_ungraceful_shutdown.service

[Unit]
Description=Log ungraceful shutdown
ConditionPathExists=!/root/graceful_shutdown
RefuseManualStart=true
RefuseManualStop=true
Before=reset_shutdown_flag.service

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/sh -c "echo $$(date): Improper shutdown detected >> /var/log/shutdown.log"

[Install]
WantedBy=multi-user.target

reset_shutdown_flag.service

[Unit]
Description=Check if previous system shutdown was graceful
ConditionPathExists=/root/graceful_shutdown
RefuseManualStart=true
RefuseManualStop=true

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/rm /root/graceful_shutdown

[Install]
WantedBy=multi-user.target

It feels like a kludge but it works pretty well. The result is I get an entry in a log file if the system wasn’t shut down properly.

CLI

Setup remote git repository with SSH & GIT

Leave a comment

I wanted to set up a simple git repository to synchronize my bash scripts between a couple hosts, no fancy github or gitlab software required. These are my notes on how I got it working. Thanks to this site for the information.

On the remote host (server)

mkdir GIT_PROJECT_DIR.git
cd GIT_PROJECT_DIR.git
git init --bare

On the local hosts (client)

Create a git repository and add files to it:

cd GIT_FOLDER
git init
git add *
git commit -m "Initial commit"
git remote add origin USER@REMOTE_HOST:GIT_PROJECT_DIR.git
git push origin master
git branch --set-upstream-to=origin/master
CLI

Modify network buffers in Linux using sysctl

Leave a comment

After installing Netdata I kept getting alerts that packets were being dropped. After some research I read you can alleviate this by increasing your system’s networking buffer. You can accomplish this with sysctl.

If you run sysctl manually it will not survive a reboot. If you want persistence you simply add the sysctl command line to /etc/sysctl.conf. Thanks to Brennen Smith on serverfault for the info on my particular problem with UDP drops:

#as root:
echo "net.core.rmem_default=8388608" >> /etc/sysctl.conf
echo "net.core.rmem_max=16777216" >> /etc/sysctl.conf
sysctl -p

The above solved my udp problem but you can modify it to change any sysctl things you want to quickly configure and persist.

Web

Rewrite HTTP host request with Varnish

Leave a comment

I had a stubborn WordPress plugin that somehow was remembering the URL of the site it was installed on. It became a problem when I changed the site URL. Despite changing the URL everywhere I could think of, this particular plugin was calling CSS files for the URL of the original site. I did a search and replace in the site database and searched all files for any reference to that site but couldn’t find anything. I never did find the culprit. My workaround was to use varnish to rewrite the request before it hit the browser.

Thanks to this answer by Jorge Nerin on Stack Overflow, I found my answer on how to do this.

backend www {
  .host = "www.example.com";
  .port = "http";
}

sub vcl_recv {
  if (req.http.host ~ "(?i)^(www.)?example.com$") {
    set req.backend_hint = www;
  }
}

In my case I had a default backend (no other backends configured) so my varnish config was simply adding these line in sub vcs_recv (varnish 4 syntax)

if (req.http.host ~ "(?i)^(www.)?old.host.name$") {
     set req.backend_hint = default;
}

That did the trick!

CLI, OS

Mount encfs folder on startup with systemd

2 Comments

A quick note on how to encrypt a folder with encfs and then mount it on boot via a systemd startup script. In my case the folder is located on a network drive and I wanted it to happen whether I was logged in or not.

Create encfs folder:

encfs <path to encrypted folder> <path to mount decrypted folder>

Follow the prompts to create the folder and set a password.

Next create a file which will contain your decryption password

echo "YOUR_PASSWORD" > /home/user/super_secret_password_location
chmod 700 /home/user/super_secret_password_location

Create a simple script to be called by systemd on startup using cat to pass your password over to encfs

#!/bin/bash
cat super_secret_password_location | encfs -S path_to_encrypted_folder path_to_mount_decrypted_folder

Finally create a systemd unit to run your script on startup:

vim /etc/systemd/system/mount-encrypted.service
[Unit] 
Description=Mount encrypted folder 
After=network.target 

[Service] 
User=<YOUR USER> 
Type=oneshot 
ExecStartPre=/bin/sleep 20 
ExecStart=PATH_TO_SCRIPT
TimeoutStopSec=30 
KillMode=process 

[Install] 
WantedBy=multi-user.target

Then enable the unit:

sudo systemctl daemon-reload
sudo systemctl enable mount-encrypted.service
OS

Linux Samba shares using Kerberos / AD credentials

4 Comments

I had a hell of a time trying to figure out why after upgrading the CentOS Samba package the samba shares quit working. Every time someone tried to access the share, the smb service would crash. I had this system configured to use active directory credentials and it worked well for a time, but no longer.

After much digging I found my problem to be the lack of a krb5.keytab file. This is due to my using PowerBroker Open instead of kerberos for authentication.

The solution was to add this line to my samba config:

kerberos method = system keytab

That one bit made all the difference. My current samba config is as follows with no more crashing: (Updated 8/29 to add workgroup name)

[global]
     security = ADS
     passdb backend = tdbsam
     realm = DOMAIN
     workgroup = NETBIOS_DOMAIN_NAME
     encrypt passwords = yes
     lanman auth = no
     ntlm auth = no
     kerberos method = system keytab
     obey pam restrictions = yes
     winbind enum users = yes
     winbind enum groups = yes

Update 8/29/2018: After updating and rebooting my smb service refused to start. It kept giving this very unhelpful message:

 ../source3/auth/auth_util.c:1399(make_new_session_info_guest)
create_local_token failed: NT_STATUS_NO_MEMORY
../source3/smbd/server.c:2011(main)
ERROR: failed to setup guest info.
smb.service: main process exited, code=exited, status=255/n/a
Failed to start Samba SMB Daemon.

I couldn’t find any documentation on this and eventually resorted to just messing around with my smb.conf file. What fixed it was adding this to my configuration:

workgroup = NETBIOS_DOMAIN_NAME

Replacing NETBIOS_DOMAIN_NAME with the old NetBIOS style domain name (what you would put in the domain part of domain\username for logging in) for my company. It worked!

CLI, Hardware, OS

Fix USB bluetooth in KDE Plasma on CentOS 7

Leave a comment

I spent too many hours trying to figure this stupid thing out.. but FINALLY! I have my bluetooth headset working in CentOS 7 with the KDE 4 Plasma environment. Read on if you dare…

First, you must configure dbus to allow your user to use the bluetooth dongle. Add the following above the closing /busconfig tag.  Be sure to replace USERNAME with your user account:

sudo nano /etc/dbus-1/system.d/bluetooth.conf
  <policy user="USERNAME">
    <allow send_destination="org.bluez"/>
    <allow send_interface="org.bluez.Agent1"/>
    <allow send_interface="org.bluez.GattCharacteristic1"/>
    <allow send_interface="org.bluez.GattDescriptor1"/>
    <allow send_interface="org.freedesktop.DBus.ObjectManager"/>
    <allow send_interface="org.freedesktop.DBus.Properties"/>
  </policy>

Remove and re-plug the adapter in.

Next, follow Arch Linux’s excellent guide on how to pair a bluetooth device using bluetoothctl

bluetoothctl
[bluetooth]# power on
[bluetooth]# agent on
[bluetooth]# default-agent
[bluetooth]# scan on

Now make sure that your headset is in pairing mode. It should be discovered shortly. For example,

[NEW] Device 00:1D:43:6D:03:26 Lasmex LBT10

shows a device that calls itself “Lasmex LBT10” and has MAC address “00:1D:43:6D:03:26”. We will now use that MAC address to initiate the pairing:

[bluetooth]# pair 00:1D:43:6D:03:26

After pairing, you also need to explicitly connect the device (every time?):

[bluetooth]# connect 00:1D:43:6D:03:26

If you’re getting a connection error org.bluez.Error.Failed retry by killing existing PulseAudio daemon first:

$ pulseaudio -k
[bluetooth]# connect 00:1D:43:6D:03:26

Finally, configure pulseaudio to automatically switch all audio to your headset by adding the following line to the bottom of /etc/pulse/default.pa:

nano /etc/pulse/default.pa

# automatically switch to newly-connected devices
load-module module-switch-on-connect

Update 7/27: I rebooted my machine and lost my bluetooth, to my dismay. I discovered that my user needs to be a member of the audio group. Since I’m in an active directory environment I think the local audio group got removed at reboot. So, to restore it, as root I had to run this:

usermod -aG audio <user>

After doing that, to prevent logging out and back in again, you can do the following:

su - <USERNAME>

Once that’s done all the bluetoothctl commands worked again.

CLI, OS, Virtualization

Backup and restore docker container configurations

1 Comment

I came across a need to start afresh with my docker setup. I didn’t want to re-create all the port and volume mappings for my various containers. Fortunately I found a way around this by using docker-autocompose to create .yml files with all my settings and docker-compose to restore them to my new docker host.

Backup

Docker-autocompose source: https://github.com/Red5d/docker-autocompose

git clone https://github.com/Red5d/docker-autocompose.git
cd docker-autocompose
docker build -t red5d/docker-autocompose .

With docker-autocompose created you can then use it to create .yml files for each of your running containers by utilizing a simple BASH for loop:

for image in $(docker ps --format '{{.Names}}'); do docker run -v /var/run/docker.sock:/var/run/docker.sock red5d/docker-autocompose $image > $image.yml; done

Simple.

Restore

To restore, install and use docker-compose:

sudo curl -L https://github.com/docker/compose/releases/download/1.21.2/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Next we use another simple for loop to go through each .yml file and import them into Docker. The sed piece escapes any $ characters in the .yml files so they will import properly.

for file in *.yml; do sed 's/\$/\$\$/g' -i $file;
docker-compose -f $file up --force-recreate -d; done

You can safely ignore the warnings about orphans.

That’s it!

Troubleshooting

ERROR: Invalid interpolation format for “environment” option in service “Transmission”: “PS1=$(whoami)@$(hostname):$(pwd)$ “

This is due to .yml files which contain unescaped $ characters.

Escape any $ with another $ using sed

sed 's/\$/\$\$/g' -i <filename>.yml

ERROR: The Compose file ‘./MariaDB.yml’ is invalid because:
MariaDB.user contains an invalid type, it should be a string

My MariaDB docker .yml file had a user: environment variable that was a number, which docker compose interpreted as a number instead of a string. I had to modify that particular .yml file and add quotes around the value that I had for the User environment variable.