Category Archives: OS

FreeNAS unable to create jails fix

I recently got a shiny new FreeNAS Mini appliance. It’s the bee’s knees. Previously I was using a virtualized instance of FreeNAS that has served me admirably for two years now. During the migration I decided to start fresh with the jails configuration I had and deleted the entire jails dataset. This turned out to be a mistake. I suddenly found out that I couldn’t create any jails or plugins. The plugin download would hang for a long time and flash a brief message “Failed to download plugin.” Not helpful.

I tried changing the location of my jails in configuration to no avail. I even tried nuking my FreeNAS config entirely and starting from scratch. The error still happened! Somehow that configuration survived a factory restore.

I finally found this freenas forum entry that pointed me in the right direction. It suggested I use the warden command to delete the plugin jail template completely and re-install it. When I tried to I got this error:

 

[nicholas@freenas ~]$ sudo warden template delete pluginjail
ERROR: Not a ZFS volume: /mnt/storage/jails/.warden-template-pluginjail

It was still trying to install the plugin template in my non-existent dataset. I decided to try re-creating the missing dataset and then running the warden delete command again. Success!

[nicholas@freenas ~]$ sudo zfs create storage/jails/.warden-template-pluginjail
[nicholas@freenas ~]$ sudo warden template delete pluginjail

Once you delete the template jail via warden, you can re-create it in the right place after configuring the correct path in Jails / Configuration. Once you have the right place configured, issue the following:

warden template create -nick pluginjail -tar http://download.freenas.org/jails/9.3/x64/freenas-pluginjail-9.3-RELEASE.tgz

Plugins and jails work again! Success.

Batch convert Global security groups to Universal

Recently I came across a need to batch convert global security groups into universal security groups in my work’s Active Directory domain. The reason for this is so I could then turn them into Mail Enabled security groups, which would enable mail to be delivered to members of these groups. Unfortunately all security groups at this organization are Global in scope.

Seeing as this is a one domain organization there is no harm in changing the scope to Universal. Doing this via mouse is very tedious; fortunately we can use a few basic command line tools to automate the task. Thanks to Jeff Guillet for outlining how to do this.

The three magic commands are: dsquery, dsget, and dsmod.

First I wanted to test out a single security group to make sure everything would work. I couldn’t convert it because it was a member of several global security groups. This rabbit hole went several levels deep. Piping together dsquery, dsget, and dsmod all together solved this problem instantly:

dsquery group -limit 0 -name "<Group Name>" | dsget group -memberof | dsmod group -c -q -scope u

The above command first gets the full name of the group specified by the -name command. The output is sent to the dsget command to query what groups that group is a member of. The output of that command is sent to the dsmod command, which does the work of actually changing each of those groups into a security group:

  • -c tells it to continue on error
  • -q tells it to not print successful changes.
  • -scope u instructs it to change the group’s scope to Universal.

Any errors will be printed to the console. Depending on how many levels of global groups there are you may have to run this command several times in order to convert the problematic groups to Universal scope.

Once that command finishes without error you can modify the group itself to be a universal group by simply omitting the middle dsget command:

dsquery group -limit 0 -name "<Group Name>" | dsmod group -c -q -scope u

After testing we are now ready to expand this to convert ALL Global security groups to be Universal in scope. If you would like a report of how many groups would be affected, run this command. It will output all groups from the query to the text file Groups.txt:

dsquery group -limit 0 | dsget group -samid -scope -secgrp > Groups.txt

To modify every group simply omit the “-name” parameter from the group command used above with our test group. This will iterate through every group in the directory and pass it on to dsmod which will modify the scope to be universal:

dsquery group -limit 0 | dsmod group -c -q -scope u

Some built-in groups can’t be converted due to their nature, so you will have to work around those (Domain Users being one example.) You will probably need to run the command a few times until no errors appear.

Profit.

 

Mount folder from another system over SSH

I recently had a need to mount a folder over SSH to allow my file manager to browse through the files on a remote system. Two great resources led me to the solution to this problem: sshfs

I first came across this little tutorial on how to install sshfs on my shiny new Linux Mint 18 box:

sudo apt-get install sshfs
sudo mkdir /mnt/droplet #<--replace "droplet" with whatever you prefer
sudo sshfs -o allow_other,defer_permissions root@xxx.xxx.xxx.xxx:/ /mnt/droplet

Pretty slick. If you want to use a keyfile instead of being prompted for a password, you can use the IdentityFile option:

sudo sshfs -o allow_other,defer_permissions,IdentityFile=~/.ssh/id_rsa root@xxx.xxx.xxx.xxx:/ /mnt/droplet

You can have this handled in /etc/fstab for automounting. Thanks to this Arch Linux guide for the info. (The command below requires systemd.)

user@host:/remote/folder /mount/point  fuse.sshfs noauto,x-systemd.automount,_netdev,users,idmap=user,IdentityFile=/home/user/.ssh/id_rsa,allow_other,reconnect 0 0

I tweaked my /etc/fstab file a bit because it complained that allow_other required a configuration change. Since I’m the only user of this box it didn’t matter to me. Here is my configuration:

nicholas@remote:/ /home/desktop/remote fuse.sshfs noauto,x-systemd.automount,_netdev,users,idmap=user,IdentityFile=/home/desktop/.ssh/keyfile,reconnect,allow_other 0 0

I’m mounting the root folder of my remote machine into a folder named remote on my desktop machine. I generated ssh keyfiles so that no password was required. Now the mount shows up under “Devices” in my file manager and a simple click mounts the folder gets me there. Sweet.

Edit 2/25/18: Added allow_other option per this article

 

Unable to update crouton after installing VirtualBox

When trying to update crouton I kept getting the following error message:

Preparing to unpack .../linux-image-3.8.11_20150314_amd64.deb ...
Ok, aborting, since modules for this image already exist.
dpkg: error processing archive /var/cache/apt/archives/linux-image-3.8.11_20150314_amd64.deb (--unpack):

After some digging I found this thread which explains how to fix it. The source of the problem was the fact that I had installed a modified kernel to allow VirtualBox to work properly. The way to update crouton is to remove the repository for that kernel

rm /etc/apt/sources.list.d/crouton-packages.list

After removing that repository the update completed successfully.

 

Persistent SSH tunnel for Windows

Over the years I’ve needed to access family members’ machines for remote support. The problem with parents and grandparents is walking them through certain prompts for services like join.me is quite problematic. To that end I’ve devised an open source way for me to automatically remote into their machine regardless of firewalls or machine location. This is possible thanks to cygwin, autoSSH, and NSSM. As long as the machine has internet access, I can get to it.

To pull this off you’ll need to install a few cygwin packages, copy over a private key file, create a batch script, and invoke NSSM to create a service to invoke the batch script on startup.

Cygwin

Obtain cygwin from here. You’ll need to use the graphical installer for the initial setup. Install the following packages:

  • ssh
  • autossh
  • wget (not necessary, but handy to have)

If cygwin is already installed, install it again. I wasted an hour once trying to figure out why it wasn’t working when the culprit turned out to be a buggy old version of cygwin itself.

Private key

For this to work you’ll need an SSH server configured for key authentication (no password.) On your SSH server:

  • Create new user for the Windows machine
  • Execute ssh-keygen as that user
  • Copy the contents of the .pub file into ~/.ssh/authorized_keys
  • Copy the private key (the one with no extension) to the Windows computer
  • Make sure permissions for the .ssh folder and everything inside of it is 600

GatewayPorts

One option that I really enjoy on my SSH server is the GatewayPorts option. This turns your SSH server into a gateway for any port forwards. Simply edit /etc/ssh/sshd_config and add

GatewayPorts yes

Save the file and restart the SSH service. Now if you create SSH tunnels your SSH server opens those ports for you to connect from other machines.

Create batch file

On the windows machine a simple command gets us up and running. Create a one-liner .cmd file on the Windows machine in a location of your choosing with the following:

c:\cygwin\bin\autossh.exe -M <random_port_number> -i <keyfile location>  -l <user> -R<remote_port:localhost:<local_port> -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null <remote address>

Update it to reflect the path of your cygwin installation if you installed somewhere other than the default location.

I add the reverse port forward option ( -R ) so that I can simply connect to my ssh server on the specified port and the connection will tunnel through to the Windows computer. In my case, I do -R5700:localhost:5900 which instructs my ssh server to listen on port 5700, then forward that connection to the Windows machine on port 5900 for VNC.

Create service

The Non-suciking service manager is a nifty little program that lets us turn anything into a windows service. Once it’s a service it can be started automatically on startup, even if nobody has logged in yet.

Obtain NSSM from here and extract it to a location you can remember. Then, open an administrator command prompt, cd to the directory containing nssm.exe, and enter the following:

nssm.exe install autossh

A GUI will open up. Specify the location of your batch file in the Path: section, then click Install service.

Once this is done, start the service by running services.msc, looking for your service, right click and select start. Make sure the startup type is set to automatic.

That’s it! If your keys are in the right place and the permissions are correct, the computer will automatically (and silently) log into your SSH server and create a tunnel for you. Autossh will continually try to re-connect in the event of connection loss. Awesome.

Reverse SSH

You can also configure cygwin to be an SSH server for your windows host. This will allow you to SSH into the machine if you specify -R<random_port:localhost:22 in your batch file. Here are a few notes for getting ssh working

  • Open up a cygwin terminal and execute the command:
    ssh-host-config
  • Once the SSH server is configured, tweak the SSH configuration to allow logging in with blank passwords (many of my family do not use a password to log into the machine.) Simply un-comment the line “PermitEmptyPasswords no” and change no to yes. Then, restart the ssh service. (thanks to this blog for the insight)

Use OpenVPN from within crouton chroot

Update: Google released a ChromeOS update that broke the openvpn script. Find the updated version here.

Update 3/29/17: Added a DNS suffix line to fix broken DNS


I posted a little while ago about how to get openvpn working on your chromebook. That guide required that you run openvpn outside your chroot on the chromebook instance itself.

Lately I’ve been really feeling the need to have openvpn run within a crouton chroot instead. The solution is to take the script from the post above and divide it into two parts. One part you will still have to run outside your chroot but you will only have to do it once on each reboot. The other part can safely live inside your chroot.

First, on your chromebook itself (not in a chroot) make a small script to tell the shill service not to kill tun0:

sudo echo "
#!/bin/bash

#Allows the tun0 device to function
sudo stop shill
sudo start shill BLACKLISTED_DEVICES=tun0" > /usr/local/bin/shill

chmod +x /usr/local/bin/shill

Next, create this script within your chroot. Be sure to modify the environment variables to suit your setup.

#!/bin/bash

CONF_DIR="/path/to/directory/openvpn/config/is/in"
CONF_FILE="FILENAME_OF_OVPN_FILE"
NAMESERVER="IP_OF_DNS_SERVER_YOU_WANT_TO_USE"
SEARCH="DNS_SUFFIX_YOU_WANT_TO_USE"

cd "$CONF_DIR"

# Add google DNS on top of current ones, since openvpn command does not do it
sudo sed -i "1s/^/# new DNS\nsearch $SEARCH\nnameserver $NAMESERVER\n# old DNS\n/" /etc/resolv.conf

sudo openvpn --config "$CONF_FILE" --dev tun0

# When ctrl-c is hit remove tun0 and cleanup the DNS
sudo openvpn --rmtun --dev tun0
sudo sed -i '/# new DNS/,/# old DNS/d' /etc/resolv.conf
trap 2

Voila, we now have openvpn working inside our chroots again.

Install Guacamole 0.9.9 on Ubuntu 15.10

Lately I’ve been trying to upgrade my installation of Guacamole 0.9.8 to 0.9.9. You’d think it would be simple. It is not. I ended up just blowing up my 0.9.8 VM and starting over, this time with Ubuntu 15.10. I found this excellent guide which got me most of the way there, with one small hiccup that took way more time than it should have to figure out.

I will paste the guide I got from the above site for convenience with my added notes for clarification on parts that I had trouble with.

Installation

#!/bin/bash
# WORKING ON UBUNTU 15.10 WITH GUAC 0.9.9 AND TOMCAT8

#Update Everything
apt-get update && apt-get -y dist-upgrade

#Install Stuff
#You will be prompted for a mysql root password. Remember this for the configuration step; change MYSQLROOTPASSWORD to whatever you enter here.
apt-get -y install libcairo2-dev libpng12-dev libossp-uuid-dev libfreerdp-dev libpango1.0-dev libssh2-1-dev libtelnet-dev libvncserver-dev libpulse-dev libssl-dev libvorbis-dev libwebp-dev mysql-server mysql-client mysql-common mysql-utilities tomcat8

# Install libjpeg-turbo-dev
wget -O libjpeg-turbo-official_1.4.2_amd64.deb http://downloads.sourceforge.net/project/libjpeg-turbo/1.4.2/libjpeg-turbo-official_1.4.2_amd64.deb
dpkg -i libjpeg-turbo-official_1.4.2_amd64.deb

# Add GUACAMOLE_HOME to Tomcat8 ENV
echo "" >> /etc/default/tomcat8
echo "# GUACAMOLE EVN VARIABLE" >> /etc/default/tomcat8
echo "GUACAMOLE_HOME=/etc/guacamole" >> /etc/default/tomcat8

#Download Guacamole Files
wget -O guacamole-0.9.9.war http://downloads.sourceforge.net/project/guacamole/current/binary/guacamole-0.9.9.war
wget -O guacamole-server-0.9.9.tar.gz http://sourceforge.net/projects/guacamole/files/current/source/guacamole-server-0.9.9.tar.gz
wget -O guacamole-auth-jdbc-0.9.9.tar.gz http://sourceforge.net/projects/guacamole/files/current/extensions/guacamole-auth-jdbc-0.9.9.tar.gz
wget -O mysql-connector-java-5.1.38.tar.gz http://dev.mysql.com/get/Downloads/Connector/j/mysql-connector-java-5.1.38.tar.gz

#Extract Guac
tar -xzf guacamole-server-0.9.9.tar.gz
tar -xzf guacamole-auth-jdbc-0.9.9.tar.gz
tar -xzf mysql-connector-java-5.1.38.tar.gz

# MAKE DIRECTORIES
mkdir /etc/guacamole
mkdir /etc/guacamole/lib
mkdir /etc/guacamole/extensions

# Install GUACD
cd guacamole-server-0.9.9
./configure --with-init-dir=/etc/init.d
make
make install
ldconfig
systemctl enable guacd
cd ..

# Move files to correct locations
mv guacamole-0.9.9.war /etc/guacamole/guacamole.war
ln -s /etc/guacamole/guacamole.war /var/lib/tomcat8/webapps/
cp mysql-connector-java-5.1.38/mysql-connector-java-5.1.38-bin.jar /etc/guacamole/lib/
cp guacamole-auth-jdbc-0.9.9/mysql/guacamole-auth-jdbc-mysql-0.9.9.jar /etc/guacamole/extensions/

Configuration

# Configure guacamole.properties 
echo "mysql-hostname: localhost" >> /etc/guacamole/guacamole.properties 
echo "mysql-port: 3306" >> /etc/guacamole/guacamole.properties 
echo "mysql-database: guacamole_db" >> /etc/guacamole/guacamole.properties 
echo "mysql-username: guacamole_user" >> /etc/guacamole/guacamole.properties
 
# This is where you will want to change "PASSWORD" 
echo "mysql-password: PASSWORD" >> /etc/guacamole/guacamole.properties 
rm -rf /usr/share/tomcat8/.guacamole 
ln -s /etc/guacamole /usr/share/tomcat8/.guacamole 

# Restart Tomcat Service 
service tomcat8 restart

#Configure the MySQL database
#Make sure you change MYSQLROOTPASSWORD and PASSWORD
mysql -u root -pMYSQLROOTPASSWORD
create database guacamole_db;
create user 'guacamole_user'@'localhost' identified by 'PASSWORD';
GRANT SELECT,INSERT,UPDATE,DELETE ON guacamole_db.* TO 'guacamole_user'@'localhost';
flush privileges;
quit

#Populate the database
#Make sure you change MYSQLROOTPASSWORD
cat guacamole-auth-jdbc-0.9.9/mysql/schema/*.sql | mysql -u root -pMYSQLROOTPASSWORD guacamole_db

After that it should be as simple as logging into your shiny guacamale server as guacadmin/guacadmin.

It wasn’t that simple for me. When I tried to log in all I got was a blank page. Reading the log file /var/log/tomcat8/localhost.<date>.log revealed the following:

Error querying database. Cause: java.sql.SQLException: Access denied for user 'guacamole_user '@'localhost' (using password: YES)

I kept changing and double checking the password for guacamole and couldn’t figure out why it was getting access denied. I even manually logged into mysql with that username and password and it worked, yet guacamole would not load in the browser.

After staring at the log long enough I realized that there is a space before the closing tick on the username. Aha! There were pesky trailing spaces in my guacamole.properties. Removing those spaces did the trick. Always some mundane detail!

Finally I have guacamole 0.9.9 working.

Xenserver NFS SR from FreeNAS VM hack

I have a Citrix xenserver 6.5 host which hosts a FreeNAS VM that exports an NFS share. I then have that same xenserver host use that NFS export as a SR for other VMs on that same server. It’s unusual, but it saves me from buying a separate server for VM storage.

The problem is if you reboot the hypervisor it will fail to connect to the NFS export (because the VM hosting it hasn’t booted yet.) Additionally it appears Xenserver does not play well at all with hung NFS mounts. If you try to shutdown or reboot your FreeNAS VM while Xenserver is still using its NFS export, things start to freeze. You will be unable to do anything to any of your VMs thanks to the hung NFS share. It’s a problem!

My hack around this mess is to have FreeNAS, not Xenserver, control starting and stopping these VMs.

First, create public/private key pair for ssh into xenserver

ssh-keygen

This will generate two files, a private key file and a public (.pub) file. Copy the contents of the .pub file into the xenserver’s authorized_keys file:

echo "PUT_RSA_PUBLIC_KEY_HERE" >> /root/.ssh/authorized_keys

Copy the private key file (same name but without .pub extension) somewhere on your FreeNAS VM.

Next, create NFS startup and shutdown scripts. Thanks to linuxcommando for some guidance with this.  Replace the -i argument with the path to your SSH private key file generated earlier. You will also need to know the PBD UUID of the NFS store. Discover this by issuing

xe pbd-list

Copy the UUID for use in the scripts.

vi nfs-startup.sh
#!/bin/bash
#NFS FreeNAS VM startup script

SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i <PRIVATE_KEY_LOCATION> -l root <ADDRESS_OF_XENSERVER>"

#Attach NFS drive first, then start up NFS-reliant VMs
$SSH_COMMAND xe pbd-plug uuid=<UUID_COPIED_FROM_ABOVE>

sleep 10

#Issue startup commands for each of your NFS-based VMs, repeat for each VM you have
$SSH_COMMAND xe vm-start vm="VM_NAME"
...
vi nfs-shutdown.sh
#!/bin/bash
#NFS FreeNAS VM shutdown script
#Shut down NFS-reliant VMs, detach NFS SR

#Re-establish networking to work around the fact that Network goes down before this script is executed within FreeNAS
/sbin/ifconfig -l | /usr/bin/xargs -n 1 -J % /sbin/ifconfig % up
SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i <PRIVATE_KEY_LOCATION> -l root <ADDRESS_OF_XENSERVER>"

#Issue shutdown commands for each of your VMs
$SSH_COMMAND xe vm-shutdown vm="VM_NAME"

sleep 60

$SSH_COMMAND xe pbd-unplug <UUID_OF_NFS_SR>

#Take the networking interfaces back down for shutdown
/sbin/ifconfig -l | /usr/bin/xargs -n 1 -J % /sbin/ifconfig % down

Don’t forget to mark them executable:

chmod +x nfs-startup.sh
chmod +x nfs-shutdown.sh

Now add the scripts as a startup task in FreeNAS  and shutdown task respectively by going to System / Init/Shutdown Scripts. For startup, Select Type: Script, Type: postinit and point it to your nfs-startup.sh script. For shutdown, select Type: Script and Type: Shutdown.

Success! Now whenever your FreeNAS VM is shut down or rebooted, things will be handled properly which will prevent your hypervisor from freezing.

 

Resizing LVM storage checklist

This is a short note of what to do when you change size of the physical disk an LVM setup, such as the default configuration in CentOS 7.

  1. Modify the physical disk size
  2. Modify the partition size
    1. I used fdisk to delete the partition, then re-create with a larger size
    2. Reboot
  3. Extend the physical volume size
    1. pvresize <path to enlarged partition>
  4. Extend the logical volume size
    1. lvextend <lv path> -l100%FREE
  5. Extend filesystem size
    1. resize2fs <lv path>
    2. #If you're running CentOS 7, the default filesystem is actually XFS, not ext4. In that case:
      xfs_growfs <lv path>
  6. Profit.

Fix inconsistent mouse cursors in Linux Mint

I love Linux Mint but a frustration of mine is the fact that the mouse cursors are inconsistent. If you change the mouse theme in the themes settings it will change for most windows, but certain windows such as chrome or wine revert to the system default mouse cursor.

I’ve finally found a fix courtesy of Ubuntu Forums. The problem lies with the x-cursor-theme being independent of the theme set in cinnamon. What you have to do is run a command to update the x-cursor-theme.

First, find the name of the mouse cursor you want from a list of your installed themes:

ls /usr/share/icons

Set an environment variable of the theme you want. Specify the folder name of the theme. For example, for DMZ-Black cursor:

CURSOR=DMZ-Black

Lastly run the command to update your cursor:

gsettings set org.gnome.desktop.interface cursor-theme "$CURSOR" && sudo update-alternatives --set x-cursor-theme /usr/share/icons/$CURSOR/cursor.theme

That’s it! You now have consistent mouse cursors. OCD demons satisfied.