My Splunk version 3.3.1 seems to have been having some issues with my SSHFS mounts – actually it was an underlying file system problem – so I decided to update to 3.4. Version 3.4.3 is out now, so I figure that the bugs in the first 3.4 release should have been fixed.

The major new features touted in 3.4 are the Windows compatbility and the addition of a light weight forwarding application. If you’re using the free license version of splunk then the forwarding application is rather meaningless anyway though, because the free license does not permit ingress of Splunk data from other servers. So in theory it’s a relatively meaningless update feature wise, I was simply hoping they had matured and optimized the code base.

The verdict? Well it looks like they have. I’ve only been running the updated version for about 12 hours, but it’s sitting on 20% less virtual memory and about 10% more real memory. What does that mean? Well the amount of memory that was commited to Splunk used to start off at about 600M and rise until plateuing at just over 1G. Committed memory is memory allocation requested by the application that is not necesserily used by the application. If it isn’t actually used by the application then the memory system can allocate more memory elsewhere. The general idea (hope) is that if the committed memory is actually called upon, then the kernel will be able to free up real memory elsewhere in order to resposnd to that real request.

In my personal opinion I think such large commitment of memory is silly. The Splunk application (3.3) was requesting 1G of memory, but only using 200M. Wha-? Committing 5x the amount of actual memory consumed?? It’s not the first time I’ve seen it, but it is the first time I’ve seen it in a (on my configuration) single threaded server application.

So now I have 220M of real memory in use by Splunk, which is fine, I guess it needs it and is doing something useful with it. It also has requested 800M of memory, so it’s still requesting just under 4x what it is using, but hey, it’s better than before! I wonder if they tuned the memory usage or just tuned something else that incidentally resulted in better memory usage…

Note that I haven’t changed anything in my configuration, I’ve just upgraded the Debian package and restarted Splunk and left it running for a while. The memory usage isn’t rising after the initial start so it seems to have stablilzed.

Oh by the way the file system problem was a result of SSHFS failing the SSH connection and not reconnecting correctly. Actually I already knew about that and I had crontab remounting the file systems every half an hour. Of course, with Splunk reading off the file system, they weren’t unmounting properly, which was also causing the remount to fail (-o remount does not seem to work with SSHFS). The solution was just to do a lazy unmount which allows the remount to work correctly (unmount -l /mnt/xxxx).