SirSpanky.com - The Secret Diary of James Pearce Aged 20-Something

I downloaded Splunk 4 a couple of months ago when it came out because I decommissioned the box that my original Splunk version was on (and changed OS’s - see my battle with FreeBSD and Splunk in earlier posts) and I needed to download the package again. At the time, there was no free license available. Actually, I didn’t realise that when I downloaded it, and I wound up with a 60 day enterprise trial license. I didn’t do anything with it that I didn’t do with the old 3.x free license, as in, I didn’t index any more data and the authentication actually just annoyed me (I’m the only user).

Anyway, it expired after 60 days a the beginning of October, and there was still no free license available, so I stopped using it. At the end of October (27th), Splunk released a free license verison. Unfortunately, they provided no instructions on going from the trial to the free license version. The marketing release mentioned that 3.x enterprise users should contact sales for upgrades, and 3.x free users should read the documentation. Unfortunately, all the documentation said was that the 3.x license wouldn’t work with 4.x, and the rest of the documentation referred to the old Splunk version. Nothing indicating how to upgrade from 3.x to 4.x free license, or trial to free license conversion.

I sent sales an email and got a response back saying “Hey, thanks for using Splunk, go read our marketing release”. I’m thinking, yeah, I did that already, that’s kind of why I emailed you… Oh well. Anyway, I’ve been trying to move all these beta monitoring servers into production,so I wanted to get Splunk working. I poked around in the directories and discovered /opt/splunk/etc/splunk.license. Renaming it so that the program couldn’t find it didn’t really work; Splunk stil said my license had expired. I wound up being ready to ditch my existing database and just install a fresh copy if necessary, but first I was going to see if I could install the new version, rip the free license file out of it manually now I knew where to find it, and put it in my old directory. Because I’m using Debian / Ubuntu, when I installed the new package version, it automatically deteted an upgrade. Ok, there goes my idea of manually copying the file, I should have gone with the tarball..however, because I’d renamed the license (or maybe it does this anyway, I’m too lazy to reinstall it now it works), the upgrade installed all of the licenses and upon the first run prompted me to agree to the license agreement for a free license. Woohoo! Working Splunk. And it kept all my data that had been indexing from the Enterprise trial.

After deciding not to use VMWare if I could find a working alternate solution, I went back and decided to actually test VirtualBox 3.0.2 on a Ubuntu 8.04 headless server. I installed FreeBSD 7.2 as a guest because it was the only ISO I had available at the time, and I will be wanting to use it in production.

My questions were:

• Does it work (i.e. the VM runs, the management interface gives me visibility and I can connect to the console)
• What is the performance like
• What is the resource hit (or “how VM’s at a time can I run on my server”)

Throughout all of this I had configured the VM with bridged networking (once I eventually got the commands for it right!) and an IDE controller for the virtual HDD.

The answer to the 1st question is yes, but the bridged networking didn’t work at first. By default, VirtualBox gave me an AMD PCNet 79C973 (it’s even present in the XML config file). FreeBSD detected this, but wouldn’t DHCP off it. The problem was that FreeBSD didn’t see the media being connected (i.e. it thought the cable was unplugged).

media: Ethernet none

I checked and rechecked (reset) all the VM settings form VirtualBox, but it looked all ok. Because VirtualBox allows different types of NIC’s, I decided to try my luck with one of the Intel NIC’s. It worked; the interface came up right away and the media was detected by FreeBSD correctly. Obviously I had to reconfigure the NIC because it was a new device, but once I’d done that, it was successfully bridged to the network. The exact command I used was:

VBoxManage modifyvm ftest -nictype1 82540EM

Which changes the NIC to an Intel Pro 1000. I really don’t care about the speed, I know it’s virtual anyway, but I do care that this one works and the AMD does not!

I think this might be a bug in FreeBSD 7+. Looking at a thread where someone is trying to get bridged mode working in the FreeNAS LiveCD, it doesn’t work in the 0.7 liveCD which is FreeBSD 7 based, but it works in the FreeNAS 0.69 CD, which is FreeBSD 6 based. Hrmm. Oh well, at least now there is a solution listed on Google somewhere!

By the way, the performance hit on a Dual P3 733 with 1.5G of RAM and nothing (I mean nothing, it was a complete fresh install of Ubuntu Server 8.04 w/ SSH enabled and VirtualBox 3.0.2 over the top) is 40% of one CPU when the VM is idle. It quickly jumps to 100% of one CPU when any disk work is done, even if the work inside the VM is not CPU intensive, so obviously virtual HDD I/O is still slow (PIO style woooooo!!).

It is, however, fast inside the VM. I’d say between 2/3 and 3/4 of native speed, depending on how much disk work is involved (disk work is about 20% of native speed). An I *really* like the VRDP facility. That is the winner for me, and the final straw which made me switch from VMWare (plus the fact that it works!

I deployed a FreeNAS server last week, and FreeNAS uses Samba for SMB / CIFS file sharing services. Everything went smoothly, but today I was informed that one of the users was having trouble Saving Excel files directly to some folders. He had been saving to his desktop and copying across, which worked. That means the folder permissions for his user were ok. What was going on? I isolated the problem to Office 2008 on his Mac OS Snow Leopard machine. I also isolated it to the fact that it was happening only when he saves as the native file formats (xlsx, docx), saving as Office 2003 files worked fine. What was doing on?

Microsoft Excel cannot access the file “server:path:to:file”
There are several possible reasons:

- The file name or path name does not exist
- The file you’re trying to open is being used by another program. Close the document in the other program, and try again.
- The name of the workbook you’re trying to save is the same as the name of another document that is read-only. Try saving the workbook with a different name.

It turns out that when saving the native format, Office 2008 will save to a temporary file, write to that file, then rename that file to the real file. The problem? When creating the temporary file, it was setting it read only. I have no idea why. I found the solution after a lot of searching on Google (I found lots of suggestions, but no solutions that worked). The solution is to disallow users to be able to change the permissions on files that are created, and to force a file creation mode.

On the share in question:

create mask = 0775
force security mode = 0775

I’ve used a Mac for quite a long time now. I used to need to diagram out database schematics and software charts, and recently I’ve gotten into business process charts. For a long time, the only tool I could find that was decent was Concept Draw. I’ve tried Omni Graffle, but found it sloppy. Concept Draw was easy to use, had the icons I wanted, and produced nice looking diagrams too. Unfortunately, it’s very expensive. Also, it ha sa major bug where sometimes when editing the text of an object, if I hit delete (backspace), it erases the whole canvas (the whole canvas actually disappears), and if I save it to save my changes, when I reopen it, the canvas is still gone. I.e. the bug is saved to the document, meaning I lose my work completely. This means that I can’t save my document when this bug occurs, and I lose the changes I’ve made since last save. It’s really annoying. Also, the program is huge, both in terms of resources and the size on the HDD.

Often I browse to look for a replacement. I’ve just discovered that X-Mind is now free. I spent about 30 minutes learning to use it, and learning to get it out of the mindset of “I only do mind maps” and have it producing nice looking flow charts. They actually look better than Concept Draw I think. Also, the relationships (connectors) between the objects (nodes) are much more flexible. The auto-routing is smarter, but when it fails, it’s very easy and intuitive to move the lines around. It uses curvey lines where appropriate too!

I started using Firefox before it was Firefox. I think it was Firebird. The main reason I started using it was the tabbed browsing. Then they introduced the ability to restore tabs on startup. So now I use tabs as short-term bookmarks. In a given work-session I might open up 30 to 40 tabs. Unfortunately, since Firefox 2, Firefox will evntually gobble up all my unclaimed RAM (usually about 2G) and an entire CPU core. The more tabs I leave open, the faster this happens. It results in a huge slow down in Firefox (5 seconds to open a new tab, 1 second delays in entering text before it avtually appears) and an eventual application crash. When it crashes it doesn’t save the tabs properly and I lose the most recent ones (about the most recent 10 or so).

Additionally, why do I have to have Google as my start page? No, I don’t want to use my current tab. No, I don’t wan to use all of my current tabs. No, I don’t want to use a bookmark. I want to use a blank page, because the whole reason that I open a tab without an address is so that I can type one in! It’s a stupid application. I hate it. I hate it marginally less than Safari (actually, I like Safari, but it is too basic for web development and provides no method for extending the capability with extensions).

I recently decdided to replace my Netgear FVX538 Modem / Router with a PC based firewall / router. The Netgear keeps crashing, dropping the ADSL link, and “forgtting” the WiFi password on reset. I’ve tried upgrading the firmware, and nothing seems to make it any better.

Anyway, the two PC distributions I decided to deploy for testing on real hardware were IPCop (1.4.20) and pfSsense (1.2.2). pfSense is incredibly slick and full featured. It really is an enterprise firewall OS in my view. The reporting options are great, and making changes / advanced configuration is sensible and easy (relative to the difficulty of performing the same changes on a normal *nix OS or a consumer firewall / modem device).

IPCop lacks a bunch of the features of pfSense, but none that affect me. The main difference is in the web interface. It’s just not as nice. It’s a little sloppier to use, and it looks a lot sloppier. It’s not terribly ugly, but it doesn’t look like something that would be produced by a commercial outfit, while pfSense does. The IPCop interface is relatively easy to edit though, it’s mainly just CGI files and a single CSS file.

There are a couple of other major function differences between the two distributions however. Firstly, pfSense is FreeBSD based (actually, based on m0n0wall), and IPCop is Linux based. I’m sure that’s the result of the practical differences that I found. What I’m talking about is the hardware support offered by the two operating systems. I had *a lot* of trouble getting pfSense to even boot from CD on Pentium hardware. Pentium 2 hardware was ok however. Even Pentium 1 MMX chips failed though. I’m sure it’s a minboard problem, as I’ve experienced the same issues with FreeBSD as well. Basically, some sort of incompatibility with the mainboard chipset causes the bootloader to fail to boot. Sometimes an immediate reset occurs, sometimes it crashes, and sometimes it fails with an error message. I tried more than half a dozen board / CPU combinations for Pentium grade hardware, and none worked with pfSense. Additionally, pfSense regused to acknowledge the presence of my Realtek chipset cards. I understand that they are cheap cards and will not offer the performance of a 3Com or Intel Pro card, but to not detect at all? That’s weird. FreeBSD has support for them, but it’s as if the kernel that pfSesnse compiled for their OS doesn’t include it. Weird.

I only tried IPCop on 2 Pentium grade systems, because that’s all I needed to try it on. I tried on a P120 and it failed to boot. I then tried it on a P90 and it worked fine. The system has a lot of RAM for a Penitum, 96MB, but it was nice to see it boot!

As to the performance, pfSense complains about having less than 128MB of RAM, and quite rightly so. It eats 64MB without doing anything. It also spikes my Pentium 2 350 CPU up to 80% load without network traffic. By spike, I mean when the OS is doing “housekeeping” activities.

On the Pentium 90 with IPCop, RAM usage didn’t exceed 32MB when not under network load, and the CPU didn’t exceed 20% usage without load. That’s a big difference; the numbers are smaller even on a ratio, so they are *much* smaller when taking into account the fact that it was running on a system that is 4x slower than the pfSense system.

What about stats for operating with load? I don’t know; I already decided to go with pfSense. Yes, it needs more hardware resources, and is more fussy, but it *is* better. I’m tired of compromising with routers. I’ve been through about 8 routers in the past 10 years, and I’m just tired. I’m 95% confident I can do everything I need to in pfSense right now, and have future support in it. With IPCop, I can probably do what I want in it now, but I’m not confident about future support, and I don’t like using the web interface.

I recently redeployed my main desktop computer to be a virtualisation hub. I’m going to run a headless VirtualBox setup on Ubuntu with it. It’ll only be using CLI. At the moment the specs of the desktop are:

• 3G DDR2
• 2.66Ghz Core2Duo
• Asus P5K
• ATI Radeon 2900XT

Now, when idle, Core2Duo’s are quite power efficient. And I’m going to be merging 3 physical boxes into this one, but none of them do continuous work (hence why I’m merging them). For some reason, when sitting idle (I haven’t installed the real OS yet, it’s just a testing version of Debian Lenny on there with no services), the machine was still spewing out heat.
It turns out that even on a CLI, the Radeon card gets quite toasty. I mean, not hot by gaming standards, but hot compared to say, an entry level card. I happened to have an entry level card around: an ATI X1300. So I thought, I’ll quickly swap them out, ensure that it was indeed the GFX card, and go merrily on my way. No such luck. The X1300 was a dud. Sometimes the PC would get to the POST if I wiggled the card, only to complain that it didn’t have a card, and other times it just wouldn’t POST at all.
Now, this is a 2008 model system. I mean, it’s early 2008, but it was top of the line. But it has PCI cards. And I figured I had a TNT2 PCI version around somewhere, or something. The “or something” turned out to be a generic PCI card. I also had an S3 trident, but I wanted to see if the generic card worked. It looked older than the S3. That means it’s about 20 years old now. I didn’t even know if recent motherboards are able to detect and boot with a PCI graphics card only. I do now. They can. Mine did. Sweeeet! It makes _no heat_. There’s not even a heat sink on the card. It doesn’t display the BIOS graphics splash screen properly (the whole “powered by Intel” image), but it displays the actual text fine, BIOS is fine, VESA images is fine (i.e. the Linux boot splash screen). It’s so cool. It also takes up a fraction of the space that the 2900XT did.

Now my system runs cool and uses less power, I’m happy[ier]

The site has been (semi) offline for a couple of days now. I took the database server that drives the site down because I’m virtualizing a lot of my infrastructure. Because it took longer than I thought it would to get everything back online, only cached pages were being served. The cache eventually times out and requires regeneration from the database, which it couldn’t do, and returned error 500 pages! Anyway, the new (virtualized) database server is up, so the site is back online

I’ve read lots of forum posts recently where someone asks how to turn encryption off for an SSH session; specifically for an SCP transfer. Every one I’ve seen has been flamed for asking this. One common response I see is:

“the encryption doesn’t take up enough of the CPU to warrant the kind of exposure on a modern CPU, you’re probably I/O limited anyway”

GARBAGE. On my Via C3 Nehmiah @ 900Mhz (it’s a 1.2Ghz chip in a 100Mhz FSB capable motherboard hence the slowed clock), my CPU taps out at 3.7MB/s on a 100Mb/s network. An it taps out on the SSH daemon, not the I/O time. Using NFS I can pull 9 to 10MB/s at CPU tap-out. Encrypting at wire speed DOES take up significant CPU time. Normal SSH terminal connections, sure, negligible. Bulk SCP connections, it’s real. Just take a look at the performance measurements taken on a Via C3 on this Linux / Via Padlock OpenSSH enabling tutorial.

Modern distributions of linux (i.e. kernel 2.6.27+ based), seem to have patched the OpenSSH (and hence the SSHD) to use the hardware encryption on the Via chip (Padlock), and I can pull 9-10MB/s at CPU tap-out on that with SCP. A P3 733Mhz also taps out at 3.5MB/s with the same Linux (Ubuntu) though, so it’s definitely the software being optimized for the Via chip.

At the end of the day though, on my local LAN (wired), I don’t really care about the encryption of the file transfer. What I care about is the ubiquity of the SSH protocol. I’m also the only one using it to access files, so I’m not using it to replace NFS, I’m just using it to access my private files (which are sometimes quite large) using the already-configured ACL (PAM). Why can’t I disable the encryption for the SSH data transfer in V2? Sigh.

I have a couple of entries that I added into my system crontab file a while back that I noticed immediately working. I’ve been running the command manually for a while because I couldn’t be bothered invetigating. When I did sit down to investigate, I found that running:

crontab /etc/crontab

Makes crontab echo the output to STDOUT. Including errors. It’s kind of annoying how those errors don’t appear in my system message log normally, but at least I found the error! The error - what looked like wordwrap wasn’t, and so crontab was seeing a syntax error