Recently I came across a situation with my home install of Splunk (free license) where the 500MB quota was exceeded three days in a row. I hadn’t checked Splunk for a few days so I was completely blindsided by it. The consequence of going over quota three days in a row? Losing the ability to do any searches in Splunk, which is a real downer.
The easiest, although least convenient, way to fix being locked out is to wait it out. If you go 30 days in a row without violating the license, Splunk will unlock itself. Splunk will still receive and index events during that time. The inability to search makes it really difficult to track down what the problem is, though, and I wasn’t happy waiting for 30 days before getting Splunk back.
Poking around on the Splunk forums I discovered that there is a way to get splunk back – perform a fresh install and then migrate your database and settings over to the fresh install. This involves backing up a few things, then copying them over the fresh install’s default folders
- $SPLUNK_HOME/var/lib/splunk/defaultdb #Default Splunk index, where all my data is held. If you have other indexes in here you’ll want to copy them too.
- $SPLUNK_HOME/etc #all your configuration files
Simply back up the above folders, install Splunk on a new machine, launch Splunk first so it will generate all the default files, then copy the files over to the new instance.
I went a step further and planned for the future. I wrote a quick and dirty script that will do all of this for you, even on the same machine – no need to copy to another machine. The script assumes you’re running a redhat derivative and have the correct Splunk install file in a predictable location. Update the locations of splunk directories and install files as needed and run as root.
#!/bin/bash #Backup important directories mkdir /opt/splunkbackup/ cp -al /opt/splunk/etc /opt/splunkbackup/ cp -al /opt/splunk/var/lib/splunk/defaultdb /opt/splunkbackup/ #Nuke splunk /opt/splunk/bin/splunk stop rm -rf /opt/splunk #Reload from fresh start rpm -iv --replacepkgs /home/nicholas/splunk-6.2.2-255606-linux-2.6-x86_64.rpm /opt/splunk/bin/splunk start --accept-license #Restore configuration files and indexes /opt/splunk/bin/splunk stop rm -rf /opt/splunk/etc cp -al /opt/splunkbackup/etc /opt/splunk/ rm -rf /opt/splunk/var/lib/splunk/defaultdb cp -al /opt/splunkbackup/defaultdb /opt/splunk/var/lib/splunk/ chown splunk:splunk -R /opt/splunk/ /opt/splunk/bin/splunk start #Remove splunk backup rm -rf /opt/splunkbackup
This will restore your searches, settings, and data. It won’t restore audit and other internal Splunk information, however. This script worked marvelously in getting my Splunk back.
6 thoughts on “Fix Splunk lockout after exceeded quota”
Splunk has a free 10GB/day license for developers. It renews every 6 months. Might be something to look into.
Thank you for this, after uninstalling and reinstalling the rpm i realized i had to blow away the configs, but forgot exactly which directories to backup, and your script saved me time as well. Glad to donate.
Thanks, I’m glad this was helpful!
Excellent, thanks! I didn’t have to follow every step, but the overall script got me up-and-running while Splunk ignored my key reset request (which should not even be necessary with the latest version).
Thank you kindly sir, it worked flawlessly
I tried to use your script but its not working for restore data, the script is restoring the configurations of my Splunk, but the data is not restoring.
I tried to restore the complete /opt/splunk/var/lib/splunk file but this one restore everything of my old splunk, including the lincese warnings.