All posts by nicholas

OS

Delete windows.old folder

2 Comments

Some time ago I upgraded my Windows Server 2012 machine to Windows Server 2012 R2. The upgrade was seamless and the server has hummed along just fine until recently, when it began running out of space.

Windirstat, a great little disk space usage reporting program, reported that the largest hog of space was the windows.old folder. Upon upgrade of the OS, the old Windows folder was renamed to Windows.old to make room for the new OS files and has sat there, untouched, ever since.

I tried to remove this folder with hilarious results. The folder is owned by TrustedInstaller. Easy enough, I’ll just replace the owner with my own user account, right? Wrong. Even after becoming the owner of the folder and everything inside it, I was prompted that I needed permission from… myself.. to delete the folder. I then tried changing the owner to “Everyone” and receive a rother comical message that I needed permission from Everyone to remove the folder. That would take some time!

everyone
You need permission from everyone.

That’s when I decided to throw in the towel and google. The solution to this problem involves the command line (thanks to here for the information.) Open an administrator command prompt and issue the following commands:

takeown /F c:\Windows.old\* /R /A /D Y
cacls c:\Windows.old\*.* /T /grant administrators:F
rmdir /S /Q c:\Windows.old

That did the trick! No more full disk.

Virtualization

Reclaim lost space in Xenserver 6.5

1 Comment

Storage XenMotion is awesome. It allows me to spin up a second Xenserver host and live migrate VMs to it whenever I need to do maintenance on my primary xenserver host. I don’t need an intermediary storage device such as a NAS – the two hosts can exchange live, running VMs directly. No downtime!

An unfortunate side effect of using Storage XenMotion is that sometimes it doesn’t clean itself up very well. It takes several snapshots in the migration process and they sometimes get “forgotten about.” This results in inexplicable low disk space errors such as this one:

The specified storage repository has insufficient space

..despite there being plenty of space.

This article explains how to use the coalesce option to reclaim space by issuing the following command:

xe host-call-plugin host-uuid=<host-UUID> plugin=coalesce-leaf fn=leaf-coalesce args:vm_uuid=<VM-UUID>

Unfortunately that didn’t seem to do anything for me. Digging into the storage underpinnings I can see that there are a lot of logical volumes hanging out there not being used:

xe vdi-list sr-uuid=<UUID of SR without space>

This revealed a lot of disks floating around in the SR that aren’t being used (I know this by looking at that same SR inside xencenter.) Curiously there is a VDI with identical names but with different UUIDs, despite my not having any snapshots of that VM.

I was about to start using the vgscan command to look for active volume groups when I got called away. Hours later, when I got back to my task, I found that all the space had been freed up. Xenserver had done its own garbage collection, albeit slowly. So, if you’ve tried to use xenmotion and found you have no space.. give xenserver some time. You might just find out that it will clean itself up.

Update 05/20/2015

I ran into this problem once more. I read from here that simply initiating a scan of the storage repository is all you need to do to reclaim lost space. Unfortunately when I ran that the scan nothing changed. A check of /var/log/SMlog revealed the following error (thanks to ap’s blog for the guidance)

SM: [30364] ***** sr_scan: EXCEPTION XenAPI.Failure, ['INTERNAL_ERROR', 'Db_exn.Uniqueness_constraint_violation("VDI", "uuid", "3e616c49-adee-44cc-ae94-914df0489803")']
...
Raising exception [40, The SR scan failed  [opterr=['INTERNAL_ERROR', 'Db_exn.Uniqueness_constraint_violation("VDI", "uuid", "3e616c49-adee-44cc-ae94-914df0489803")']]]

For some reason one of the ISOs in one of my SRs was throwing an error – specifically a Xenserver operating system fixup iso, which was causing the coalescing process to abort. I didn’t care if I lost that VDI so I nuked it:

xe vdi-destroy uuid="3e616c49-adee-44cc-ae94-914df0489803"

That got me a little father, but I still wasn’t seeing any free space. Further inspection of the log revealed this gem:

SMGC: [7088] No space to leaf-coalesce f8f8b129[VHD](20.000G/10.043G/20.047G|ao) (free
 space: 1904214016)

I read that if there isn’t enough space, a coalesce can’t happen on a running VM. I decided to shut down one of my VMs that was hogging space and run the scan again. This time there was progress in the logs. It took a while, but eventually my space was restored!

Moral of the story: if your server isn’t automatically coalescing to free up space, check /var/log/SMlog to see what’s causing it to choke.

Analytics

Automatically delete old data in Splunk

Leave a comment

I’ve had Splunk humming along for about two years now. I’ve already increased the storage space for my Splunk VM once. Today I received a notice that I’ve once again run out of space and indexing had been suspended. I wanted a more permanent solution to this problem, so I consulted the almighty Google.

The solution to my problem is to set a retirement policy. This allows Splunk to automatically delete old data when you hit a certain index size. You can also go by time, but I opted for size. It’s a pretty simple configuration change. Simply edit (or create if it doesn’t exist) $SPLUNK_HOME/etc/system/local/index.conf and add two lines

sudo vim /opt/splunk/etc/system/local/indexes.conf
[main]
maxTotalDataSizeMB = 50000

Then, restart your Splunk instance (the command below is Debian/Ubuntu specific)

sudo service splunk restart

The configuration above tells Splunk to keep at most 50GB of data. When that limit is reached, it begins deleting the oldest log files. No more out of space errors!

Gaming, OS

Install ventrilo on Ubuntu 14.04 64bit

1 Comment

Ventrilo is a voice communication server which is popular in the gaming community. It allows teams of people to get together and have voice chats. I recently tried to install vent on a 64bit instance of Ubuntu 14.04. When I tried to execute the server binary, I was greeted with this lovely error message:

bash: ./ventrilo_srv: No such file or directory

It’s a pretty cryptic error message that had me chasing my tail for a while until I came across this post which shed further light on the issue. This error stems from trying to run a 32bit binary on a 64bit system without the proper libraries installed.

A simple

sudo apt-get install lib32z1

Resolved this issue. After those 32bit libraries were installed, vent ran without issue.

CLI

Using screen to run interactive programs at startup

1 Comment

Oftentimes I will encounter programs that weren’t necessarily designed to be automatically run that I want to run on startup. Sometimes that program will have interactive information that you will want to see later, but you still want it to run on startup.

The solution to this particular problem is using screen in combination with su and bash. In my situation, I want to run the HDSurfer plugin on bootup as a different user. The solution I came up is as follows (thanks to superuser.com and stackoverflow.com for the guidance I needed to set this up.)

Install screen

Screen is like having a separate X window session to keep a program running, except it is for console programs. You can attach and detach to this screen whenever you’d like and not worry about the program terminating.

sudo apt-get install screen

Create a script to run your program with all required arguments

In my case I needed to execute the command “python /usr/bin/HDSurferWave/hdsurferwave.py start” as a different user in a screen session (so it wouldn’t terminate when the terminal session did.) To do this,

  • invoke screen with the -dm command (to begin the program in detached mode)
  • issue the bash -c argument afterward to invoke bash
  • Include your desired command after that

My one line script looks like this:

screen -dm bash -c "python /usr/bin/HDSurferWave/hdsurferwave.py start"

Run your script

I use the su command with the -c argument to change the user that will be running the script, as the startup script launches things as root by default (with pre-systemd systems, anyway.) The -s command initiates a shell to launch, and the last argument is the user you want to run as. My launch argument is:

su -c "/usr/bin/HDSurferWave/start.sh" -s /bin/sh nicholas

Configure the script to run on startup

Edit /etc/rc.local and add your script command from above, then mark that file as executable by running chmod +x /etc/rc.local. Note: This will not work with systems using systemd.

 

 

Web

Disable access logging in Tomcat 7

2 Comments

Guacamole is a great HTML5 VPN gateway. It allows me to access internal applications without having to install any software. I wrote about it briefly in this article.  It wasn’t until I noticed that my Splunk indexer reported warnings that I had exceeded my 500MB quota (the free license maximum amount) that I realized that guacamole has a verbosity problem.

In examining the logs it appears that Guacamole passes about 6 HTTP requests per second while you’re using it. This problem is magnified if you have guacamole sitting behind an apache server, as each request is logged twice – once in Apache access logs, and again in Tomcat access logs.

Since I already have that same information in apache access logs and I don’t allow access directly to Tomcat, I set out to disable Tomcat logging completely. Things have changed between versions so it got a little confusing.

To disable logging in Tomcat 7, you have to edit /etc/tomcat7/server.xml (that’s where it lives in Ubuntu Server 14.04 anyway) and comment out a section (thanks to Stack Overflow for helping me figure this out.)

vim /etc/tomcat7/server.xml

Find this line:

    <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"  
           prefix="localhost_access_log." suffix=".txt"
           pattern="%h %l %u %t &quot;%r&quot; %s %b" resolveHosts="false"/>

Comment out the line like this:

    <!-- <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"  
           prefix="localhost_access_log." suffix=".txt"
           pattern="%h %l %u %t &quot;%r&quot; %s %b" resolveHosts="false"/> -->

Save the file and restart Tomcat.

:wq
service tomcat7 restart

No more duplicate logging.

OS

Re-enable Windows Defender in Windows 8.1

Leave a comment

I recently underwent an exercise in frustration – taking a stock Lenovo G50 laptop and trying to get it into a workable state. Like many lower end laptops it came with a lot of bloatware, not the least of which was Mcafee Antivirus-premium-whatever garbage.

When I removed Mcafee and rebooted, I got a notice that Windows Defender was not enabled. When I clicked to enable it… nothing happened. I tried several times to no avail. I tried manually starting the service and it failed. I tried hard core things like running sfc /scannow and even editing the registry per here, but the issue remained.

Finally I came across this article which showed me how to fix it – simply launch windows defender by hitting start and searching for Windows Defender. Click the big red “Turn On” button. That’s all it takes! Why you can’t do that through the action center or by manually starting services is beyond me. Amazing.

Note: it’s possible that the combination of editing the registry, rebooting, and then launching the Windows Defender program fixed the issue. Whatever – it works now 🙂

Analytics

Get geolocation info in Splunk with iplocation

Leave a comment

Splunk 6 has many awesome new features, one of which is built-in IP geolocation. No longer do you have to manually lookup up city, state, and country when investigating logs – Splunk will do that for you. This page has the details.

For example, if I want my x_forwarded_for IP addresses to have geolocation, I tack this at the end of my query:

| iplocation x_forwarded_for | stats count by x_forwarded_for City Region Country

The fields iplocation can produce are:

  • City
  • Continent
  • Country
  • lat
  • lon
  • MetroCode
  • Region
  • Timezone

You can combine this query with DNS lookups (as detailed here) for a more complete picture of your data.

<search query> | iplocation x_forwarded_for | lookup dnslookup clientip as x_forwarded_for OUTPUT clienthost as hostname | stats count by x_forwarded_for City Region Country hostname

Neat.

Analytics

Extract multiple Active Directory fields in Splunk

2 Comments

I had posted here about how to extract account names with a specific modifier (exclude account names ending in a dollar sign.) That worked for one specific instance, but I found I needed something better. Active Directory logs have multiples of the same value (Account_Name, Group_Name, etc.) that all depend on context, namely the value of the line two lines above it.

For example,

Message=A member was added to a security-enabled universal group.

Subject:
 Security ID: <Random long SID>
 Account Name: Administrator
 Account Domain: ExampleDomain
 Logon ID: <random hex value>

Member:
 Security ID: <Another random long SID>
 Account Name: CN=George Clooney,OU=ExampleDomain,OU=Hollywood,OU=California,DC=USA,DC=NA,DC=Terra

Group:
 Security ID: <Yet another long SID>
 Account Name: Old Actors
 Account Domain: ExampleDomain

You can see that there are three different Security ID fields, three different Account Name fields, and two different Account Domain fields. The key is the context: Subject account name, member account name, or group account name.

I wrestled for some time to find a regex expression for Splunk that would continue matching things after a line has ended. After much searching I came across this post which explained the need for a regex modifier to do what I wanted.

In my case I needed to use the (?s) modifier to include newline characters in my extraction. My new and improved AD regex extraction is as follows:

(?s)(Group:.+Account Name:\s+)(?P<real_group_name>[^\n]+)
  • (?s)  Regex modifier indicating to include new lines
  • Group:  Section I am interested in. You can replace this with Member: if you’re interested in member account names instead
  • .+ match one or more of any character (including new line as indicated by modifier above)
  • Account Name:\s+ This is in conjuction with the previous two items to create a match that includes the section name and anything after that until the spaces after Account Name
  • [^\n]+ Match one or more characters that is not a new line (since you might have an account name with spaces.)

Finally! This is the regex I’ve been looking for.

 

Hardware, OS

Fix tiny text in Windows 8.1

Leave a comment

I really enjoy my new Microsoft Surface Pro 3. It has a high DPI screen which makes things very clear and sharp. Unfortunately, when you plug it into an external monitor, many Windows applications don’t deal with the DPI setting properly and thus appear blurry and/or the text is very tiny.

The workaround for this issue is a new compatibility mode setting in Windows 8 – Disable display scaling on high DPI settings. Simply right click on the shortcut of the problem application and go to properties, then go to the Compatibility tab, then check the box.

dpi

Success. Thanks to Microsoft for the information.