Category Archives: CLI

CLI, Web

Rotate videos for WordPress using ffmpeg

Leave a comment

WordPress has an extremely annoying issue with reading EXIF data when posting photos and videos. There is a plugin to fix rotated pictures, but I couldn’t find one for rotated videos. If you happen to upload a video from your phone that was shot in portait orientation, it will likely upload to wordpress in landscape orientation (sideways.) Very annoying.

My fix to this is to re-encode the video so it is at the proper orientation before uploading to wordpress. Thanks to this site for the information – ffmpeg does this automatically for you. So the syntax is simple:

ffmpeg -i SOURCE_VIDEO_FILENAME -c:a copy FIXED_VIDEO_FILENAME.mp4

Success.

CLI, OS, Web

Active Directory / LDAP integration with WordPress

2 Comments

I struggled for a while to get WordPress to use Active Directory credentials on CentOS 7. Below is how I finally got it to work.

First, install necessary packages:

sudo yum -y install openldap-clients php-ldap

If you use self-signed certificate for ldaps, you’ll need to modify /etc/openldap/ldap.conf

HOST <HOSTNAME_OF_LDAP_SERVER>
PORT 636
TLS_CACERT <PATH_TO_CA_CERT>
TLS_REQCERT demand

With the above settings you can test your ldap string with ldapsearch

ldapsearch -x -D "<BIND USERNAME>" -b "<BASE_DN>" -H ldaps://<LDAP_SERVER_HOSTNAME> -W sAMAccountName=<USER_TO_QUERY>

Once ldapsearch works properly, install your AD integration plugin.  I use AuthLDAP by Andreas Heigl

I struggled with which LDAP strings and filters to use. This is what finally got everything working with our Active Directory environment:

LDAP URI: ldaps://<BIND_USERNAME>:<BIND_PASSWORD>@<AD_SERVER_ADDRESS>:636/<BASE DN>

Filter: (sAMAccountName=%s)

Name-Attribute: givenName

User-ID Attribute: sAMAccountName

Second Name Attribute: sn

Group-Attribute:memberOf

Group-Separator:  _

Group-Filter: (&(objectClass=user)(sAMAccountName=%s)(memberOf=*))

Role – group mapping

I had to change Group-Separator to _ above, because in Role – group mapping for active directory, you must put the FQDN, which includes commas. Put an underscore separated list of FQDNS for each of these fields you want.

CLI, OS

Using expect with the Ansible shell module

3 Comments

In one of my ansible playbooks I need to obtain a file from a Windows share. I can’t find a module that handles this so I’m using the shell module to call the smbclient command to do what I need. The problem with this solution is that smbclient prompts for a password (and I don’t want to supply it on the command itself for security reasons.)

I tried using ansible’s built-in expect module, but frustratingly it only works on systems that have pexpect >= 3.3  , which CentOS 7 & Ubuntu 14.04 do not have.

My solution to this is to install the expect command on the host, and then use the ansible shell module to call it, following the example given in Ansible’s shell module page

Part of the process in my playbook is registering stdout from that command for later use. I then ran into a problem where I would run smbclient -c “ls <filename>” but ansible would register nothing. After some digging I found I also need to include the interact command after sending the password. Without it, anything after sending the password is not registered to stdout. Thanks to rostyslav-fridman on Stack Overflow for the answer.

My final problem was I was sending a password that had a ] character in it. It was causing this error on run:

missing close-bracket\n while executing\n\"send

I found here (thanks glenn-jackman) it was due to  the fact that the expect syntax uses tcl language, which treats those brackets as special characters. To get around this I had to use an ansible filter, specifically regex_escape()

Lastly I ran into an issue specifically with how I was spawning smbclient. I kept getting this message:

"stderr": "send: spawn id exp4 not open\n while executing\n\"send

It boiled down to single vs double quotes. If I put my -c arguments in single quotes it failed; with double quotes, it worked.

My completed play is below. Finally, success!

- name: Get RSA filename 
  shell: |
    set timeout 300
    spawn smbclient {{standards_location}} -W DOMAIN -U {{username}} -c "ls {{file_location}}*.tar"
    expect "password:"
    send {{ password | regex_escape() }}\r
    interact
    exit 0
  args:
    executable: /usr/bin/expect
  changed_when: false
  no_log: true
  register: RSA_filename_raw
CLI, OS, Virtualization

Gaming VM with graphics passthrough in Arch Linux

1 Comment

At one point I had KVM with GPU passthrough running in Arch Linux. I have since moved away from it back to ProxMox. Here are my notes I jotted down when I did this in Arch. Sorry these are just rough notes, I didn’t end up using Arch for long enough to turn this into a polished article.

pacman -Sy qemu netctl ovmf virt-manager

When creating VM, make sure chipset is Q35

CPU model host-passthrough (write it in)

Create VirtIO SCSI controller and attach drives to it

NIC device model: virtio

—- networking —-

Create bridge:

https://wiki.archlinux.org/index.php/Bridge_with_netctl

Copy /etc/netctl/examples/bridge to /etc/netctl/bridge

/etc/netctl/bridge
Description="Example Bridge connection"
Interface=br0
Connection=bridge
BindsToInterfaces=(enp4s0)
IP=dhcp

#Optional - give your system another IP for host-only networking
ExecUpPost="ip addr add 192.168.2.1/24 dev br0"
sudo netctl reenable bridge
sudo netctl restart bridge

In the VM add another network interface, also assign to br0. Manually specify IP in guest VM to match subnet specified above in ExecUpPost

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

Allow UEFI bios: https://wiki.archlinux.org/index.php/libvirt#UEFI_Support

sudo vim /etc/libvirt/qemu.conf

/etc/libvirt/qemu.conf
nvram = [
    "/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"
]

sudo systemctl restart libvirtd

Edit VM hardware:

CLI: sudo virsh edit <vm name>

GUI: double click on VM, then click second icon fnom the left (little i bubble)  Add GPU this way

Nvidia GPU: need to do x otherwise code 43

<features>
	<hyperv>
		...
		<vendor_id state='on' value='whatever'/>
		...
	</hyperv>
	...
	<kvm>
	<hidden state='on'/>
	</kvm>
</features>

Hot add CD:

sudo virsh attach-disk <VM_NAME> <ISO LOCATION>  hdb –type cdrom

Add second NIC: https://jamielinux.com/docs/libvirt-networking-handbook/bridged-network.html

sudo virsh edit win10

<interface type="bridge">
   <source bridge="br1"/>
</interface>

CPU configuration

Current Allocation 16

Topology / manually set CPU topology

1 socket, 16 cores, 1 thread

<cputune>
<vcpupin vcpu=’0′ cpuset=’16’/>
<vcpupin vcpu=’1′ cpuset=’17’/>
<vcpupin vcpu=’2′ cpuset=’18’/>
<vcpupin vcpu=’3′ cpuset=’19’/>
<vcpupin vcpu=’4′ cpuset=’20’/>
<vcpupin vcpu=’5′ cpuset=’21’/>
<vcpupin vcpu=’6′ cpuset=’22’/>
<vcpupin vcpu=’7′ cpuset=’23’/>
<vcpupin vcpu=’8′ cpuset=’24’/>
<vcpupin vcpu=’9′ cpuset=’25’/>
<vcpupin vcpu=’10’ cpuset=’26’/>
<vcpupin vcpu=’11’ cpuset=’27’/>
<vcpupin vcpu=’12’ cpuset=’28’/>
<vcpupin vcpu=’13’ cpuset=’29’/>
<vcpupin vcpu=’14’ cpuset=’30’/>
<vcpupin vcpu=’15’ cpuset=’31’/>
</cputune>

Running Windows 10 on Linux using KVM with VGA Passthrough

 

 

--machine q35 \
--host-device 4b:00.0 --host-device 4b:00.1 \

https://medium.com/@calerogers/gpu-virtualization-with-kvm-qemu-63ca98a6a172

 

add usb ports. Doesn’t work if nothing’s in the port?

virsh edit win10

<hostdev mode=’subsystem’ type=’usb’ managed=’yes’>
<source>
<address bus=’3′ device=’2’/>
</source>
<address type=’usb’ bus=’0′ port=’2’/>
</hostdev>

Remove Tablet input device to get 4th USB passthrough option

 

 

 

Troubleshooting

internal error: Unknown PCI header type '127'

https://forum.level1techs.com/t/trouble-passing-though-an-rx-580-to-an-ubuntu-desktop-vm/123376/3

Threadripper PCI Reset bug: https://www.reddit.com/r/Amd/comments/7gp1z7/threadripper_kvm_gpu_passthru_testers_needed/

Error 43:

<features>
	<hyperv>
		...
		<vendor_id state='on' value='whatever'/>
		...
	</hyperv>
	...
	<kvm>
	<hidden state='on'/>
	</kvm>
</features>

Audio cuts out whenever microphone is used

I had a very odd issue where all sound disappeared in my Windows VM if the microphone was used. Even simply opening up audio properties and going to the Recording tab triggered this issue. Disabling / re-enabled Special Effects for the playback device brought it back until the microphone was accessed again.

I’m using USB sound card passed through to the VM for audio. It stems from the VM’s USB controller. When I had it set to USB3 the issue would occur. When set to USB2 the issue went away. Bizarre.

CLI, OS

Add Ubuntu PPA key to Debian

Leave a comment

Occasionally I want to install packages located at an Ubuntu PPA repository on my Debian stretch machine. There’s a bit of a trick to it, thanks to chrisjean.com for outlining what needs to be done.

Step 1 is the same as in Ubuntu, add the PPA with add-apt-repository (install if it’s not already there)

sudo add-apt-repository ppa:<contributor>/<ppa name>

This will appear to work but when you do an apt update you may get something similar to this

W: GPG error: http://ppa.launchpad.net/jonathonf/gcc-7.1/ubuntu xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8CF63AD3F06FC659
W: The repository 'http://ppa.launchpad.net/jonathonf/gcc-7.1/ubuntu xenial InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.

The last step is to manually import the key with the following command:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys <KEY_OF_PPA>

The PPA key will be listed on the PPA’s page. Once I ran that second command everything worked swimmingly.

CLI, OS

Flatten nested AD group memberships with powershell

2 Comments

Several applications at my job do not know how to read nested security groups. This is annoying because we grant everything through security groups instead of individual entitlements.

I’ve recently finished writing a powershell script that will “flatten” a security group that has nested security groups. This script reads a security group’s membership, compares the individually assigned users with the nested security group membership, and then reconciles them so only members of the nested security group are individually added to the main group. It allows me to simply add a security group to another security group, and still be able to use the group to grant access to applications that don’t support nested groups. It also ensures that nobody has rogue access they shouldn’t have. Everything managed through groups like God intended.

I consulted a ton of different sites to accomplish this. Here are just a few:

https://www.reddit.com/r/PowerShell/comments/3f7iki/flatten_out_active_directory_groups_containing/

https://stackoverflow.com/questions/11526285/how-to-count-objects-in-powershell

https://stackoverflow.com/questions/41658770/determining-object-type

https://docs.microsoft.com/en-us/powershell/module/activedirectory/

https://ss64.com/ps/syntax-compare.htmlhttps://ss64.com/ps/compare-object.html

#Nested Security Group flattener script
#Written by Nicholas Jeppson, 10/6/2018

#This script scans nested security groups and compares their membership to that of the base security group.
#It then reconciles membership so that the only members of this group are those who are members of the nested security groups.
#This is required for applications that cannot read nested security groups, such as mattermost.
#No more manually adding people to a group after you've already added their role to that same group!

#=============Variables section=============#

#Enter groups to reconcile here, separated by quotes and a comma:
$groups_to_flatten = @("group1","group2")

#==========End Variables Section=============#

#Loop through each group to flatten
foreach ($group in $groups_to_flatten) {

    Write-Host "`nProcessing group ""$group"""

    #Read current individually added users
    $individually_added_users = get-ADGroupMember -Identity $group | Where-Object {$_.objectClass -eq 'user'}

    #Read group membership of nested groups - Ignore specific user (optional)
    $nested_group_members = get-ADGroupMember -Identity $group | Where-Object {$_.objectClass -eq 'group'} | Get-ADGroupMember -Recursive | Where-Object {$_.name -ne 'USER_TO_IGNORE'}

    #Compare current individually added users with that of nested security groups
    $users_to_add = Compare-Object -ReferenceObject $individually_added_users -DifferenceObject $nested_group_members -PassThru | Where-Object {$_.SideIndicator -eq "=>"}
    $users_to_remove = Compare-Object -ReferenceObject $individually_added_users -DifferenceObject $nested_group_members -PassThru | Where-Object {$_.SideIndicator -eq "<="}
    
    #loop through each user to remove and remove them
    foreach ($user in $users_to_remove) {
        Remove-ADGroupMember -Identity $group -Member $user -Confirm:$false
        Write-Host "Removed: $user"
    }
    
    #loop through each user to add and add them
    foreach ($user in $users_to_add) {
        #Add nested group membership individually back to the parent group
        #Write-Host "Adding individual members to ""$group""`n`n"
        Add-ADGroupMember -Identity $group -Members $user -Confirm:$false 
        Write-Host "Added: $user"   
    }
}
CLI

Powershell equivalent of “find -exec”

Leave a comment

I recently found myself on a Windows 10 system needing to do the equivalent of “find . -name *.mdi -exec mdiconvert -source {} -log log.txt \;” I knew what to do instantly on a Unix system, not so on a Windows system

I finally figured it out and am now writing it down because I know I’ll forget! Thanks to these several sites for pointing me in the right direction.

  • Get-ChildItem is the find equivalent
    • -Filter is the -name equivalent
    • -Recurse must be specified otherwise it only looks in the one directory
  • % is an alias for “ForEach-Object
  • Put the command you want run in brackets {}
  • Put an ampersand in front of the command you wish to run so you can properly pass arguments containing dashes
  • $_.FullName turns the powershell object into a text string (which my command required.) FullName is the full path of the item found with Get-ChildItem
    • $_ is the rough equivalent of find’s {} (the item that was found)

The command I ended up using is below (find any .mdi files and use Microsoft’s mdi2tif utility to convert the result to .tif files)

Get-ChildItem "C:\Users" -Recurse -Filter *.mdi | % { & 'C:\Program
Files (x86)\modiconv\MDI2TIF.EXE' -source $_.FullName -log log.txt }
CLI, Hardware, OS

Make FreeDOS boot ISO to flash BIOS

1 Comment

I needed to flash the BIOS of one of my old server motherboards and to my dismay found the only way to do so was via DOS boot image. It was not straightforward so I thought I’d write it down. Thanks to pingtool & tummy.com for the information I needed to pull it off.

First, obtain a copy of FreeDOS ISO and extract it to a directory

  • mount -o loop <freedosISO.iso> <mount directory>
  • rsync -aP <mount directory> <directory you want files to copy to>

Next, copy the necessary flash utilities and firmware files to that same directory as above.

Lastly, use genisoimage to create a new ISO image based on the above folder. Modify -o output to wherever you want the ISO to go.

sudo apt install genisoimage
cd <folder you copied your files to>
mkisofs -o /tmp/freedos_biosupdate.iso -q -l -N \
   -boot-info-table -iso-level 4 -no-emul-boot \
   -b isolinux/isolinux.bin \
   -publisher "FreeDOS - www.freedos.org" \
   -A "FreeDOS beta9 Distribution" -V FDOS_BETA9 -v .

From here you can take the ISO and mount / burn it as needed. It will boot into FreeDOS. Tell it to go to a shell and away you go.

CLI, OS, Virtualization

FreeNAS ZFS tuning for SSDs

Leave a comment

I wanted to optimize my all SSD storage array on my FreeNAS server but I had a hard time finding information in one place. After a lot of digging I pulled things from several places. This is what I came up with. It boiled down to two main settings

  • ashift
  • recordsize

Checking ashift on existing pools

zdb -U /data/zfs/zpool.cache | grep ashift

I read here a recommended setting of ashift=13, recordsize=8k for VM workloads on SSDs.

How to change recordsize:

This is easily done in the GUI or command line and can be changed on the fly.

zfs set recordize <value> <volume>

How to change ashift:

Backup your data and destroy the pool.

Modify the setting dictating minimum ashift setting as outlined here

sysctl vfs.zfs.min_auto_ashift=13

Re-create the pool.

Note: this lustre wiki actually recommends ashift 12 instead of 13.

UPDATE 2/26/20: You can also set the ashift at pool creation time as dictated here

zpool create POOL_NAME -o ashift=12 ...

Additional reading

http://open-zfs.org/wiki/Performance_tuning#Alignment_shift
https://www.reddit.com/r/zfs/comments/7pfutp/zfs_pool_planning_for_vm_storage/

CLI, OS, Virtualization

Free up RAM after Proxmox live migration

Leave a comment

I ran into an issue where after migrating a bunch of VMs off of one of  my hosts, the remaining VMs on it refused to turn on. Every time I tried the command would hang for a while and eventually error out with this message

TASK ERROR: start failed: command '/usr/bin/kvm -id <truncated>... ' failed: got timeout

I suspected this might be due to RAM use and sure enough the usage was too high for a system that didn’t have any VMs running on it.  I found here that I could run a command to flush the cache:

echo 3 > /proc/sys/vm/drop_caches

That caused the RAM usage to go down but the symptom of the VM not starting remained. I then saw the KSM sharing still had some memory in it. I decided to restart the KSM sharing service:

sudo systemctl restart ksmtuned

After running that the VM started!