SirSpanky.com - The Secret Diary of James Pearce Aged 20-Something

spanky@zen:~$ ssh user@host
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
30:82:36:ff:f4:78:e9:c9:df:94:8b:41:35:f6:2b:a4.
Please contact your system administrator.
Add correct host key in /home/spanky/.ssh/known_hosts to get rid of this message.
Offending key in /home/spanky/.ssh/known_hosts:5
RSA host key for [host]:22 has changed and you have requested strict checking.
Host key verification failed.
spanky@zen:~$

When you update OpenSSH server, timetimes it will change the host fingerprint. This means you will find yourself getting the above error on any host that you have saved the host fingerprint on. The fingerprint is designed to verify that you are indeed connecting to the host that you say you are. As the warning suggests, it can detect man-in-the-middle attacks. However, if you know that you are getting this because you updated software (or you have regenerated the host fingerprint manually), you probably just want to be able to connect to the server again!

You may already know that host fingerprints are stored in ~/.ssh/known_hosts. The problem can sometimes be that the individual entries are not stamped by host. Well, they don’t need to be, they are unique entries. So how do we know which one to delete? The line:

Offending key in /home/spanky/.ssh/known_hosts:5

Actually tells us that the key is indeed stored in ~/.ssh/known_hosts, AND it is on line 5. If you open up the file in vi, you will probably find each it looks like there are many more than 5 lines. Don’t worry, it’s just wrapping the line. Just count down to line 5 (alternatively, use :set numbers to see the line numbers - press the colon key, then type “set numbers” and hit enter). Then hit “dd” to delete the line”. :wq (colon then “wq”) to save the file and exit, and you should be sorted!

I recently came into possession of a brand new Dell 56k Modem. These are the little external USB modems they ship with newer laptops when you request a modem, because the laptops themselves don’t come with one built-in. On the back of the modem the model specifics are labeled as “Conexant RD02-D400, REV A03, NW147″. Conexant are a chip maker for modems.

When I first plugged the modem into my Mac, it detected as a modem, but it didn’t know what it was (by default it was set to an Apple modem), and it powers the modem (one of the lighs on the modem is for power, the other is for a connection). I dug around on Google for some information on configuring it with Mac OS X. I found a post from 2005 describing a manufacturer called Zoom that uses Conexant chipsets in their modems. Zoom *is* on the Apple modem list. Zoom even have an installation instruction manual for Apple OS X Leopard on their support site! I wonder if it’s coincidental that the Dell modem looks very similar to the Zoom 3095 modem? Hmm…At least it works now though!

Kelly Calrson is Kimber on Nip / Tuck - haha, Kimber dolls, that plot line cracked me up…oh sorry back to the post!

Anyway yeah she’s hot yadda yadda, anyone who’s watched the show knows that. So I watched this movie called Made Of Honor earlier in the week and she’s in it. Actually shes kind of just a side character…she’s only in it for a wedding…as the bride. My gosh the dress she’s wearing is hot though. That movie was released last year, and Nip / Tuck has been going for 4 years I think, and Kelly Carlson is still hot. That’s a pretty good effort I think. So anyway I was thinking as well, she’s the only girl / model / actress that I think looks hot with short hair. I mean that in the broadest possible way, completely outside of hollywood stars I’ll never meet…I can’t ever remember meeting a girl with hair above the shoulder and thinking ‘oh wow, she’s just gorgeous, I wish I had a girl like that in my life’.

Now, when I was searching for some pictures for the post, I came across the photos from the Stuff magazine shoot that she did a couple of years back (can’t miss ‘em when you search for her on Google Images). And there’s a couple of nice ones with short hair…and there’s a couple of her with long hair huh? Is her hair just put up? It doesn’t look put up to me, it looks short. Ah well I don’t know. I don’t even know what those pink lace ribbon things coming out of the teddy she is wearing are for

Now it was really hard to find any pics of her looking at hot as she does in Nip / Tuck or even in the Medal Of Honor movie (I don’t think I’ll marry a girl who wears that kind of wedding dress…but it sure looks good on her!!). So I took the best I could find for Medal Of Honor - pinched from justjared.com (with a little editing), and two from the Stuff magazine shoot (well, a very air brushed collection of it!) - the one I like best with her having her hair short (up??), and the one with the pink ribbon things that I don’t know what they are for

Monit is used for tracking service availability and taking action when services are detected to have failed. One action it takes is sending an email. Of course, you can configure the options of the message sent, the subject, and the apparant sender. Now, the manual for monit isn’t quite correct on this it seems. I copied the manual sectino for mail-format syntax into my monitrc file, only to have monit fail and tell me there was a syntax error on the mail-format!

It seems the problem is that mail-format may not use line breaks without having a command. The manual has a format similar to the following, there there is a break between the opening bracket and the first command:

set mail-format {
    from: monit@my.host.com
    subject: $SERVICE $EVENT at $DATE
    message: Monit $ACTION $SERVICE at $DATE on $HOST,
        $DESCRIPTION

    Yours sincerely,
     Monit @ $HOST
}

This fails on a syntax error. Removing the line break like this:

set mail-format { from: monit@my.host.com
    subject: $SERVICE $EVENT at $DATE
    message: Monit $ACTION $SERVICE at $DATE on $HOST,
        $DESCRIPTION

    Yours sincerely,
     Monit @ $HOST
}

Makes it work (tah-dah!).

On a another note, I think this blog has had too many technical articles of late. I’ll try to post something not related to computers soon!

I broke my Splunk server by accidentally deleting the Linux libc6 libraries, and it ended up being easier to install FreeBSD than to fix it. I wasn’t going to reinstall Linux…with Splunk available for FreeBSD, why would I do that?

The install went fairly well after installing the compat6x port. Splunk say that the software works with 6.0 “or higher”. Thats a white lie. It doesn’t work natively on 7.0 (yet).

Anyway I got it running, started configuring it, and all seemed sweet. Had it index a bit of data and what not all good. When I started adding in the sources from my other servers, things went weird. Ok there was about 600M of logs, but I black listed well over half that. I figured I might go over the 500M limit of the free licence while importing everything but oh well. Anyway the server kept churning at 100% CPU usage for about 5 hours. This took me over midnight, which allowed me to see how much data had been indexed. Apparently it indexed 8G, which is really weird when there is not 8G of log files.

I tried fine tuning the black list to remove more files and limiting the time stamp information that was getting collected, but I couldn’t make Splunk finish indexing. I knew it was indexing because that’s the only thing other than searching that really ramps up the CPU usage in Splunk…it’s a fairly single minded application. I watched the splunkd log file for a while and couldn’t see anything too wrong. I wound up editing splunk/etc/system/log.cfg and setting category.FileInputTracker=WARN to info like I’d done before to see what phantom files it was indexing. It turns out there were 2 files it was getting stuck on. One was the original Debian installer log, which is about 2300 lines or so. The other was the syslog.0 file, which was about 5000 lines. What it looks like in the splunkd.log is this (the change in CRC on the splunkd.log file is because of all the info being pumped out to it):

01-19-2009 06:26:17.906 INFO  FileInputTracker - Computing CRC for seekPtr=5d188000 filename=/mnt/spanky_log/syslog.0
01-19-2009 06:26:17.911 INFO  FileInputTracker - Computing CRC for seekPtr=939298 filename=/usr/local/splunk/splunk/var/log/splunk/splunkd.log
01-19-2009 06:26:17.982 INFO  FileInputTracker - Computing CRC for seekPtr=5d190000 filename=/mnt/spanky_log/syslog.0
01-19-2009 06:26:17.988 INFO  FileInputTracker - Computing CRC for seekPtr=93939d filename=/usr/local/splunk/splunk/var/log/splunk/splunkd.log
01-19-2009 06:26:18.060 INFO  FileInputTracker - Computing CRC for seekPtr=5d198000 filename=/mnt/spanky_log/syslog.0
01-19-2009 06:26:18.068 INFO  FileInputTracker - Computing CRC for seekPtr=9394a2 filename=/usr/local/splunk/splunk/var/log/splunk/splunkd.log
01-19-2009 06:26:18.139 INFO  FileInputTracker - Computing CRC for seekPtr=5d1a0000 filename=/mnt/spanky_log/syslog.0
01-19-2009 06:26:18.145 INFO  FileInputTracker - Computing CRC for seekPtr=9395a7 filename=/usr/local/splunk/splunk/var/log/splunk/splunkd.log
01-19-2009 06:26:18.221 INFO  FileInputTracker - Computing CRC for seekPtr=5d1a8000 filename=/mnt/spanky_log/syslog.0
01-19-2009 06:26:18.226 INFO  FileInputTracker - Computing CRC for seekPtr=9396ac filename=/usr/local/splunk/splunk/var/log/splunk/splunkd.log
01-19-2009 06:26:18.295 INFO  FileInputTracker - Computing CRC for seekPtr=5d1b0000 filename=/mnt/spanky_log/syslog.0
01-19-2009 06:26:18.300 INFO  FileInputTracker - Computing CRC for seekPtr=9397b1 filename=/usr/local/splunk/splunk/var/log/splunk/splunkd.log
01-19-2009 06:26:18.371 INFO  FileInputTracker - Computing CRC for seekPtr=5d1b8000 filename=/mnt/spanky_log/syslog.0
01-19-2009 06:26:18.376 INFO  FileInputTracker - Computing CRC for seekPtr=9398b6 filename=/usr/local/splunk/splunk/var/log/splunk/splunkd.log
01-19-2009 06:26:18.446 INFO  FileInputTracker - Computing CRC for seekPtr=5d1c0000 filename=/mnt/spanky_log/syslog.0
01-19-2009 06:26:18.451 INFO  FileInputTracker - Computing CRC for seekPtr=9399bb filename=/usr/local/splunk/splunk/var/log/splunk/splunkd.log
01-19-2009 06:26:18.523 INFO  FileInputTracker - Computing CRC for seekPtr=5d1c8000 filename=/mnt/spanky_log/syslog.0
01-19-2009 06:26:18.529 INFO  FileInputTracker - Computing CRC for seekPtr=939ac0 filename=/usr/local/splunk/splunk/var/log/splunk/splunkd.log
01-19-2009 06:26:18.599 INFO  FileInputTracker - Computing CRC for seekPtr=5d1d0000 filename=/mnt/spanky_log/syslog.0
01-19-2009 06:26:18.604 INFO  FileInputTracker - Computing CRC for seekPtr=939bc5 filename=/usr/local/splunk/splunk/var/log/splunk/splunkd.log
01-19-2009 06:26:18.686 INFO  FileInputTracker - Computing CRC for seekPtr=5d1d8000 filename=/mnt/spanky_log/syslog.0
01-19-2009 06:26:18.696 INFO  FileInputTracker - Computing CRC for seekPtr=939cca filename=/usr/local/splunk/splunk/var/log/splunk/splunkd.log
01-19-2009 06:26:18.784 INFO  FileInputTracker - Computing CRC for seekPtr=5d1e0000 filename=/mnt/spanky_log/syslog.0
01-19-2009 06:26:18.790 INFO  FileInputTracker - Computing CRC for seekPtr=939dcf filename=/usr/local/splunk/splunk/var/log/splunk/splunkd.log

And it did that pretty much ad infinitum. The seekPtr DID go up, but it never completes the file. How do I know it wasn’t just taking it’s time? Because it did a whole bunch of files larger than 5000 lines each in about 4 seconds. An hour for the syslog file didn’t make sense. I deleted the syslog.0 file, so I don’t know wha the deal was there. I did keep the Debian installer log file that did the same thing, but I’m not in touch with any of the developers so it’ll probably just sit on my HDD. At least on this blog Google will pick it up for other people to know! By the way, don’t forget to change log.cfg and set category.FileInputTracker back to WARN else you will fill up your splunkd log file with self-replicating entries!

Oh yeah, another note, in FreeBSD 7.0, Splunkd will not show up with the correct CPU usage. It will say 1 to 9% on mine, but the system says there is 99% in user. if you run:

top -IS

You will see only the processes using CPU. I know splunkd isn’t displayed properly because it’s the only process displayed as running when there is 99% user!

EDIT 26th-Jan: After running Splunk over the weekend and watching the issues, I’ve discovered that it DOES come out of the loop eventually, it just re-reads the file many times. Case in point, I had a test Icecast server turned on, but it wasn’t doing anything, so the /var/log/icecast/stats.log file was 606 bytes…Splunk had indexed 202 MEGABYTES of it (found by looking at the index stats dashboard plugin). I’m also exceeding my quota many times because of these problems that crop up…the easiest way I can see to avoid them is still just to keep an eye on the index and just delete them as you see them.

I use SSHFS which uses FUSE. On FreeBSD, I generally compile core components with ports. On attempting to compile fusefs-sshfs, I got an error with fusefs-kmod:

fusefs-kmod  requires the userland sources to be installed. Set SRC_BASE if it is not in /usr/src

Actually the error is fusefs-kmod-xxxxxx for the version. It’s a pretty well known error. I figured it needed the kernel sources. I was wrong (it actually says kernel source if you’re missing that). What it wants, is some of the source for the FreeBSD software. The next question was which software does it want the source for? The answer is “mount” which is in “sbin”. So if you’re getting this error, do a sysinstall and install the source packages for base, sys and sbin and you’ll be right. No need to install ALL userland source like the message implies!

One project I have had recently is to compile a live CD with the open source flight simulation game FlightGear. Since Windows has licensing issues, the obvious choice for me was to go Linux. Linux also lends itself nicely to Live CD’s as people have been doing the development for about 10 years now (Remember the first edition of Knoppix!!!). Morphix was a commonly used platform a couple of years ago, but it seems not much has been done in the way of maintenance of the actual program or documentation since the end of 2006. This seriously inhibited my ability to get a LiveCD working the way I wanted it to with Morphix. Specifically, it annoyed me having to split main modules and mini modules as they never seemed to operate the way they were supposed to on the actual live CD.

I have used Slax in the past to build rescue CD’s, but I really would prefer something that is based on the Debian style of package management (Morphix is based on Knoppix which is based on Debian (phew!)). In my research I stimbled upon the Debian Live movement, an official movement by Debian to make Debian LiveCD’s. The tools that they make available seemed unwieldly at first, but that’s just because the man pages are kind of bad. With the help of lots of hard disk space and a free license for VMWare Server, I was able to figure out how things worked. And work it does, very well.

Ok let’s bring this back to the topic heading. Great, I can make a maintable live CD based on Debian and port FlightGear into it. The problem is, getting proper 3D acceleration. I don’t use 5 year old hardware. The oldest hardware I’ll be running this on is 18 months. That means nVidia 6xxx hardware at the oldest, and Radeon 5xx hardware. There isn’t very good support for that in Debian Stable (Etch - 4.0). My next project was thus to import the ATI and nVidia proprietary drivers into the CD. The way I found to do this was to copy the driver setup package onto the live CD, boot into the LiveCD, compile the libraries, then copy the libraries off the disc and remaster the disc with the actual compiled libraries. This actually worked and I was able to insert the kernel module. However, the X11 module for ATI did not seem to want to talk to the Kernel module even though it inserted correctly, which meant that 3D acceleration was still missing. Also, the nVidia GLX module in X11 kept inserting automatically when it was present, stopping the Mesa or FGLRX module from inserting. Sigh. In other words, it seems that it was going to be a pain to get working and a pain to maintain.

Ok here’s where the gaming gets interesting. Because I was searching for Linux LiveCD information and getting proprietary 3D drivers to work, I discovered 2 live CD distrobutions that are built explicitly for this purpose. live.linuX-gamers.net and the Supergamer live CD. live.linuX-gamers.net is built around open source games, but really does showcase some good native Linux games that have been built. Supergamer is built around a showcase of games that were available commercially for Windows and have been made free for Linux in one form or another, and so really showcases what a LiveCD can do. And it does appear to use the proprietary drivers. If I had the master build files, I’d strip the games out of it and put FlightGear in, but as it is it’s just the ISO that’s distributed and I don’t want to spend time in email conversations to get information…if they don’t want to make it publically documented at this point then I’ll just wait. It’s still a great effort though (see this great review with screenshots)!

I’ve recently been playing with Linux LiveCD’s for a project I have. I was originally using Morphix, but it seems that is so out dated that no resource is ‘complete’. The morphtools kind of worked, but it was incredibly difficult to specify temporary directories to the point where I would have to hack the scripts up directly.

Anyway long story short I did a survey of what else was out there to build Linux LiveCD’s and specifically Debian LiveCD’s. I found that Debian have an official project for this. It’s not as user-friendly as some of the other LiveCD tools perhaps, but I know it will last the test of time because that’s what Debian is good at.

So I’ve been playing around with the live-helper applications, and I stumbled upon an error. Since Google didn’t help me, I thought I’d post it.

If you perform an lh_build in a particular directory, it creates a .stage directory. Because this is a dot (.) file, it doesn’t show up in the listing. There are a variety of files in this directory that store the current status of the build. If something happens in the build, the status may not be cleared properly, and it ‘locks’ the build. This means if you try to perform the build again, you get something like this happening:

$ sudo lh_build
P: Begin caching bootstrap stage…
P: Begin bootstrapping system…
W: skipping bootstrap
P: Begin caching bootstrap stage…
W: skipping bootstrap_cache.save
P: Begin caching chroot stage…
P: Begin mounting /dev/pts…
E: system locked

Note that it says the system is locked. The solution is to delete the .stage directory. I didn’t bother examining the contents of the directory because all I was interested in was keeping the cache. Now Google will have something on this error if anyone else has the problem though.

In my professional capacity I have a client who is hosting a site on a VPS. The site is basically structured as a large blog / forum and it receives many (in the millions) page views each day. The database is approximately 200MB. I originally had the site hosted ona 256MB VMWare VI3 VPS. Needless to say it thrashed the virtual memory because there was not enough RAM. RAM on a VMWare VPS (or any VPS) in Australia is expensive, so I shifted it off site and trebled the RAM: 768MB with Linode. The site has continued to run into problems with the load scaling way above what it should given that there is so much CPU and Memory free: The load consistenly reaches up to 2. It’s a quad core server so in theory anything up to 4 is “fine”, but if every site this size put this level of hardware up to that load then I think server prodivders would be making a lot more money than they do!

After working on some home-server projects and re-analysing the performance metrics I have of the server / site, I’ve come to the conclusion that the server is I/O bound on the database site. MyTop shows lots of sleeping processes: They can’t be waiting on CPU because the CPU is 80% idle, and the memory has 200M free. The I/O is operating at about 6MB/s read and 2MB/s write. That’s seriously low for even a consumer SATA drive. I really expected better from such a large Linode plan. It seems I’m not the only person to have had this problem on Linode.

I have (again in the professional sense) some sites on a server through Slicehost and I’ve not run into I/O problems. That said, I don’t have any sites on the Slicehost servers of this magnitude. I’m wondering if it will be just as bad. If it is, then I/O may become a deterrent for me to use VPS. I wonder how many other people have had to make the shift from virtual to dedicated because of I/O bottlenecks? I’m hoping not, judging by my own internal tests and what some other people are saying.

While I’m on the topic of Slicehost, I have to say, I’m REALLY impressed with their customer service. I’ve only had a couple of problems since I’ve been dealing with them (a couple of years now) and they have always gone way beyond what I would consider their duty of care in order to help me. Compare that the Linode (about a year of service): Linode are helpful, but they don’t give any support out of the ordinary. If you’re asking questions about something they don’t officially support, then they don’t try to help. Slicehost does

TorrentFlux (actually TorrentFlux B4rt is what I’m using) is a PHP / AJAX Bitorrent front end that uses the Java Azereus BT client to actually connect into the network. I use it because this way I can run BT on one of my little home servers that is on 24/7 rather than having to leave my huge desktop computer on 24/7 or using my laptop for BT.

One problem I’ve always had with TorrentFlux / Azereus is the amount of memory it consumes. I have a feeling that is a side effect of using a Java BT client: There is probably a lot of connection table caching going on. Don’t have a solution to that one (yet). Another problem I’ve had with it is that it’s /slow/ compared to normal desktop BT clients. Sure, I expect it to be a little slower but I mean it’s slllllloooooowww. When adding a new file it takes 2 - 3 minutes to download it and another 90 seconds to sart it (it says “processing” for that time). My original solution to this was to improve the performance of the database: I shifted from using SQLite to using MySQL. I found that bogged down the server I run it on too much, so when I rolled out a dedicated MySQL box, I moved the database over to that. Well, now it didn’t bog down the server with the BT client; it bogged down the MySQL server. The weird thing is, with the performance monitoring data I had, I couldn’t figure out where the slow down was coming from. The CPU wasn’t tapping out, nor the memory. What I found was that hte I/O was thrashing.

I ran MyTop for a little while to examine the queries on the TorrentFlux database and discovered that the database gets hammered with status updates about the files that are in the queue. The specific queries are always of the form:

SELECT xxxx FROM tf_log WHERE file=’xxxx’

So obviousl I checked the details of the tf_log table. It’s a huge table holding hundreds of thousands of records an almost a hundred meg. I checked the indexes and suprisingly found that file was not one of the indexes. No wonder it’s so slow and the I/O is thrashing; every time a WHERE query is performed it’s having to do table scans! What I did was create an INDEX for the file column in the tf_log and now my TorrentFlux is about 5x faster when adding / removing / pausing / resuming downloads. The exact query you would need to issue to MySQL would be:

CREATE INDEX ON tf_log (file(20));

I’m limiting the index to 20 characters of the file name because I don’t generally have many files with similar names in my list. This way the index is kept relatively small compared to the full filename size but still having a high probablity of being unique i.e. more efficient.