KernelCrash

Sometimes I think I spend too much time on ebay. I’m still using my Thinkpad R60 as a home server;  running with Scientific Linux 6.2, using it as a media server plus KVM VM hosting. But I’m always on the lookout for something (cheap)  to replace it, as it only goes to 3GB of ram … which is a pain when you start to run a few VMs on it.  So for reasons unknown I recently bid on a Thinkpad T410 on ebay, thinking that surely I would be outbid, and that I really didn’t want one anyway (;-)… but somehow I now have a Thinkpad T410.

The auction was a bit of a lucky dip, as the guy selling it seemed to omit the specifics of which T410 it was (again, a bit of madness on my part for bidding without bothering to ask). But I ended up with a i5-560M model with 4GB of RAM, 160GB 7200rpm drive, 1280x800LCD. $350 for that was OK I thought. Given brand new laptops are so cheap these days, you might think I overspent, but there are some key things I was looking for that are sadly missing on a lot of brand new ‘low end’ laptops;

• A CPU with EPT support. My latest interest is ‘Nested Virtualisation’. All VMware’s base products (ESXi, Workstation, Player, Fusion) now support nesting, but you have to have a CPU that does EPT. All my other systems at home are core2duo or older and there is no core2duo with EPT support. It’s only in some Xeon’s and most of the i3/i5/i7 stuff.
• Can expand up to 16GB of ram. I actually got this one wrong as a T410 really only goes up to 8GB , hence why I felt like an idiot when I won the auction. But 8GB is OK. I can jam a few VMs into that, and shoving an SSD in the T410 partially compensates for the inability to go to 16GB
• Lots of disk expansion options. A key one is jamming a hard drive into the DVD bay. Sadly, the Ultrabay’s that I have that work on the R60 and my old T42 don’t work in the T410. But it is  about $10 on ebay to get a ‘Ultrabay Slim’ that does work in the T410. The T410 also has Firewire 400 (which I do actually use) and an esata port, and a 34mm expresscard slot plus the obligatory USB 2.0 ports. So there’s some options there so far as adding more disks.

So far I’ve swapped out the 160GB drive with a Intel 330 120GB drive. It really flies. A quick ‘hdparm -tT’ on the drive hits about 240MB/s (the T410 is only SATA II) which is plenty good for me. I’ve put Fedora 17 on it for now (and made sure I am not using GNome 3 or KDE!), and just playing around with some KVM VMs on it. Originally I was trying to use thin volumes for each VM, but I kept on hitting 100% on my thin pool which then ends up with a few thousand ‘requested block’ errors in /var/log/messages. So, until I can figure out how to do auto extension on a thin pool, I’ve been using VMs on an ext4 fs using qcow2 images. They work pretty well these days. You can do ‘instant’ disk level qcow2 snapshots now so long as you pick to use ‘external’ snapshots. However, you still can’t expire or rollback these external snapshots easily.

So I am liking the T410 for the most part. Quick pros/cons after having it for a few weeks;

Pros

• Fast
• Cheap
• Great keyboard (but never as good as older Thinkpads)
• Lots of disk expansion options
• Reasonably easy to get Linux running OK on it (the main things for me are; the fan is generally quiet, the graphics chipset can be configured OK, and I can suspend/resume)

Cons

• The palm rest area is a little too warm for my liking. Definitely walmer than most macbooks I’ve used.
• The Intel HD graphics requires a bit of effort to set up properly
• Only goes to 8GB of ram.
• battery life is pretty average

 

I still use my 2008 Penryn white Macbook as my day to day laptop, but I am still on OSX Snow Leopard on it as I am pretty disillusioned with where Apple is going with OSX. Lion just seemed worse than Snow Leopard, and Mountain Lion even worse (and I’d have to buy a new Mac just to run Mountain Lion). Snow Leopard is great for my needs. I really like it. I always say, for my main day to day computing system I need a “portable unix workstation that does not suck”. When I bought the Macbook, I had previously spent a year or so trying to use a linux based laptop for everyday home and business use …. and well I eventually gave up as it was too hard. Macbook plus Leopard was great in comparison. Macbook plus Snow Leopard was a little better. But these new OSXs are for some other target market.

But could I use Linux on a laptop again in a day to day basis? I’m not sure. In some ways, some of the new stuff in many linux distros is just as odd as the direction OSX is taking. eg. Gnome 3? Having to download some additional software to change the font size on the desktop is just silly. And KDE just looks a bit too weird for me.  So far I just have a minimal X setup on Fedora 17. I don’t even use gdm or similar. I just log in on the console, and type ‘startx’ to start a fluxbox session. I don’t even like fluxbox, but its easy to install on Fedora and is not as painful as Gnome or KDE.

Just some notes about sort of newish stuff (depends on your distro) in Linux LVM. It’s August 2012. I’m mainly using Redhat 6.3/Fedora 17 at the moment.

Snapshot Merging into Original (ie. drop the snapshot and revert to the pre-snapshot state)

If you dig through my old posts you’ll see some convoluted method of doing this using low level device mapper commands. But now you can do it with a simple command. First let’s say you create a snapshot:

  lvcreate -s -n lvsnap -L1G /dev/vgcrap/lvhome

Then make some modifications to whatever is in lvhome, and you decide you want to ‘roll back’ to what is in the snapshot. So you just do this;

  lvconvert --merge /dev/vgcrap/lvsnap

Cool stuff. What I thought was cooler is that it even works on mounted volumes with one caveat. Here’s what happened when I tried to do this on a root logical volume;

  Can't merge over open origin volume
  Merging of snapshot lvsnap will start next activation.

So basically, if you reboot now, when the LVM config is first loaded it merges the snapshot back in then (which logically is a safe time to do it).

I’n not too sure when snapshot merging came into the more stable distros. I don’t think Redhat 6.1 had it, but it’s definitely there in 6.3

Auto extension of snapshots

(NB: Apologies if this has been in LVM for ages. I only just noticed it). One of the semi annoying things when creating snapshots of logical volumes is that you need to ‘guess’ how much “changes”  you expect to occur during the life of the snapshot. So in my example earlier I used ‘-L1G’ to allow for up to a gig of changes. I think you’ve always been able to manually extend the size of a snapshot, so I could have written a job that regularly runs ‘lvs’ and looks for snapshots that are getting close to 100% full, then does a ‘lvextend’ on the snapshot volume to ‘give it a bit more space’ depending on the space left in your VG.

Well it turns out there are some config params in /etc/lvm/lvm.conf that will do this auto extension for you. For example, if I have a snapshot get to 70% full, I can auto-extend it by another 20%

 snapshot_autoextend_threshold = 70
 snapshot_autoextend_percent = 20

One thing I immediately noticed when I tried this out myself is that it is still a ‘delayed’ detection. ie. if your snapshot fills up too quickly and hits 100% before the autoextend detector kicks in then you’re stuffed. My test case was with a very small snapshot, and logically as you use larger snapshots at the outset, the time it takes to jump from say 70% to 100% takes longer anyway.

Thin Pools and Thin Volumes

If you have used any kind of storage appliance or ESXi or Xenserver you will know all about ‘thin volumes’. The basic premise is that if say you create a logical volume of say 100GB, you might not be using all of it initially. Perhaps you are using it for a VM, and the initial install only takes 2GB or so, so that’s 98GB that you have sort of tied up for a rainy day. The idea behind thin volumes is that you only really use space out of a logical volume when you write to a block of it. Unwritten blocks are considered to be zeroed, and so if you read from the other 98GB (in my example) you just get zeroes from the LVM driver. No disk read occurs. This way ‘that other 98GB’ can be used for other volumes.

Anyway, You can do this with lvcreate now. You need to create a thinpool first. So you still have to set aside a bit of disk space out of your VG first, but you can make it on the small side and either manually extend it or (just like with snapshots) you can configure in lvm.conf to auto-extend when it reaches a certain size (or so I thought). Anyway, let’s say I want a thin pool called ‘thinpool’ of 90GB in size

  lvcreate -T -L 90G /dev/vgcrap/thinpool

So that ‘ties up’ 90GB that you cannot use for regular volumes in your VG.

Then you create a thin volume within your thin pool;

  lvcreate -T -V 100G -n lvthintest /dev/vgcrap/thinpool

Notice how the size is now specified with ‘-V’ (for virtual size) and notice how the size can be bigger than your thin pool.  You just have to remember, that disk space is only used as you write to it. You can now create a file system on your thin volume, write stuff to it, and the filesystem will ‘think’ it is on a 100GB partition. Of course, the thin volume will only keep working so long as the thin pool does not fill up. So don’t dd /dev/zero to the entire device. Even though you are writing zeros, they are still ‘writes’ so you will eventually fill the thin pool before you fill the thin volume itself.

Just like with regular logical volumes, you can make snapshots of thin volumes, and the cool thing is that you do not have to specify a size.

lvcreate -s -n lvsnap2 /dev/vgcrap/lvthintest

Regarding the auto-extend functionality of thin pools, well I could not get it to work (so far I’ve tried Redhat 6.3 and Fedora 17). The details of how to use it are in the comments of /etc/lvm/lvm.conf on most distros, for whatever reasons, it never auto-extends for me.

 thin_pool_autoextend_threshold = 70
 thin_pool_autoextend_percent = 20

Another very cool feature of thin volumes is that they can actually shrink themselves under certain circumstances. Basically what you do is create a thin volume, put an ext4 file system on it and mount the ext4 filesystem with the ‘discard’ option just like how you would mount it if you were using an SSD drive with TRIM support. What the discard option actually does is to pass ‘delete’ requests to the next block layer down. In the case of an ext4 filesystem on a standard partition on an SSD drive, the ‘deletes’ might get passed to the kernel SATA driver, and hence the ‘deletes’ are passed to the SSD drive. With my example, the ‘deletes’ are now passed to the underlying LVM layer, and LVM knows that it can shrink a thin volume if it receives appropriate ‘delete’ requests. This is regardless of whether you are using an SSD or not. In fact, I think the ‘deletes’ are passed all the way down the chain, so if you were using an SSD, these TRIM requests also end up being sent to the SSD.

One area where this ‘pass down’ of deletes does not appear to be implemented is in qemu-kvm. So if you say create a thin volume, then use that thin volume as the virtual disk that qemu-kvm uses, and let’s say that you install linux in this VM and have its root fs as an ext4 filesystem with ‘discard’ set on the mountpoint, then it doesn’t appear to pass the ‘delete’s down any further than qemu-kvm. The net result is that your thin volume grows and grows (I did figure out a terrible way around this; stop your VM, use kpartx on the thin volume to create some partitions in the host OS such that you can then see the root OS partition from your guest. Then you can run ‘e2fsck -f -E discard /dev/vg/thinvol’, remove the kpartx devices, then start your guest up again).

Better LVM RAID devices

If you’ve used other logical volume systems like Veritas or the built in one with HP-UX, you would have noticed that software RAID is built right in. For example, if you created a simple logical volume with no redundancy, it was a simple logical volume related command to dynamically add a mirror in. You could dynamically break mirrors, mount then elsewhere etc. All very very useful. Compare that with the common usage cases for LVM on Linux, and a quick google will see that you either layer LVM on top of a hardware RAID controller solution, or you layer it on top of MD style Linux Sofware RAID. In fact there are lots of recipes for creating RAID1 or RAID5 redundancy using the md driver, then using these md devices as PV’s in a volume group.

However, LVM on Linux has seemingly had some kind of software RAID ability for some time … but if you bothered to read the docs on it, you’d come away thinking it was a bit odd. For example, the older style LVM RAID1 wanted you to use three disks. Have a look at this redhat page on the older style LVM mirroring. I’m sure quite a few admins have gone ‘well I’ve got my two drives, and I want to create a mirror, let’s check the docs online … and WTF? I need a third disk????’ If you dig a bit, you’ll see some people put this ‘extra log’ data on one of the two disks in a mirror (and hence you don’t need a third disk), or they put this ‘extra log’ in memory which means that your entire mirror needs to resync everytime your box reboots!

So I was quite pleasantly surprised when going through the Redhat 6.3 doco to discover this ‘RAID Logical Volumes’ page. So now there is a ‘raid1′ type as well as the older ‘mirror’ type. Strangely the LVM commands will still let you create the older style mirrors with their ‘extra log’ or the newer ‘raid1′ ones that work the way you want them to work (ie. you use up two disks and you can lose either disk).

You can also split a RAID1 mirror into two new logical volumes as well. I think there is even some clever stuff to allow you to track changes so that a split mirror can be merged back together again later without having to do a full resync.

You can also do RAID 4, 5 or 6 with LVM and admitedly I have not tested them out.

One thing that I tried to do with the new raid1 type is get GRUB2 to boot off it. I’ve been using Fedora 16 and 17 a bit lately and even though GRUB2 seems relatively annoying it has the great feature of being able to boot your kernel off a logical volume. That means you don’t need a separate /boot anymore. BUT, it does not work with these new ‘raid1′ type LVM volumes. So that means you still probably need to put /boot on an md device

NetworkManager is one of those things in the linux world that many SysAdmins hate. It’s a bit like your mother-in-law; it means well but is really really irritating. On any new system I set up, the first thing I do is get rid of it, and just use the old-school config files way of setting up your networking. As an experienced sysadmin, my mind is always trying to think ahead to ‘how is someone going to troubleshoot this some time in the future’. Config files and ifconfig and route commands are pretty easy and well understood. NetworkManager is something that appears to make things easier but if it doesn’t quite work the way you want it can be a bucket of pain.

Which leads me to this post. I was helping to set up a Fedora 16 system recently that had already been installed … and given that NetworkManager is the default, the sysadmin who set it up left it using NetworkManager. The thing is they had left out the DNS nameserver settings and I needed some name resolution. Of course the easy solution is edit /etc/resolv.conf and add in a nameserver line and in less than 30 seconds (assuming you have the network connectivity in place) you have name resolution. This is what I did and of course it worked and I was happy.

But then there was a power failure at the site and the server rebooted. It came up OK and the first thing I noticed was that /etc/resolv.conf did not have a nameserver entry in it. I knew straight away “Oh, that’s NetworkManager overwriting /etc/resolv.conf on boot”. So what to do to fix it? The normal answer is “Hop on the console, go to whatever Network preferences is called this week and add in the DNS IP to the Wired Connection in NetworkManager”. Unfortunately I did not have console access to this server. I did have a remote ssh connection so I tried logging in with x forwarding on, checked that I could launch an xterm, and tried running nm-applet .. which just hung and did not launch the GUI. I also sudo’d to root, made sure all my xauth and DISPLAY was right, checked that I could launch an xterm, then tried nm-applet … and again ‘nothing’. The thing is I had kind of seem this behaviour before on redhat-ish systems … and come to the conclusion that “PolicyKit is evil”.

So what next? Surely there must be a command line interface to NetworkManager? Nope. The closest thing is nmcli, but it has the great non-feature of not letting you change much. Sure it can give you some great status info, let you enable and disable some stuff, but no-can-do if you want to just update the DNS server.

Some googling later, and I’m finding some info about using dbus-send to query and control NetworkManager. In the examples below each dbus-send command gives you a little bit more info for typing in the next command (eg. the first one returns /org/freedesktop/NetworkManager/ActiveConnection/0 on my system for the list of active connections)

#work out the active connections
 dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager \
/org/freedesktop/NetworkManager org.freedesktop.DBus.Properties.Get \
string:org.freedesktop.NetworkManager string:ActiveConnections

#get the list of devices
 dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager \
 /org/freedesktop/NetworkManager/ActiveConnection/0 \
org.freedesktop.DBus.Properties.Get \
string:org.freedesktop.NetworkManager.Connection.Active string:Devices

#get the list of IP4Config stuff
 dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager \
/org/freedesktop/NetworkManager/Devices/1 \
org.freedesktop.DBus.Properties.Get \
string:org.freedesktop.NetworkManager.Device string:Ip4Config

# print the nameservers. I knew this was the right command as I
#tried it on a test system that already had a DNS server assigned
 dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager \
/org/freedesktop/NetworkManager/IP4Config/0 \
org.freedesktop.DBus.Properties.Get \
string:org.freedesktop.NetworkManager.IP4Config string:Nameservers

OK now, so I could drill down to get the nameserver detail. Great. Surely there is some easy way to then set it? After more googling it eventually transpired that the IP4Config stuff I was getting was a read-only dbussy thing. And that there must be some other way to actually ‘write’ to the settings. After more sleuthing (and using the GUI tool d-feet), I worked out this command;

dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager \
/org/freedesktop/NetworkManager/Settings/0 \
org.freedesktop.NetworkManager.Settings.Connection.GetSettings

That prints out a heap of stuff including my IP address and DNS info, but its all in UINT32 type format, so an IP address like 192.168.0.100 is shown as;

uint32 1677764800

So how did I figure out that ‘Settings/0′ bit? You can get a list of the settings available with;

dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager \
/org/freedesktop/NetworkManager/Settings \
 org.freedesktop.NetworkManager.Settings.ListConnections

OK, now I thought there must be a way of using dbus-send to update the DNS address. I tried a few different ideas that all failed, and then found some stackoverflow comment indicating in the dbus-send man page how ‘D-Bus supports more types than these, but dbus-send currently does not’. Great.

So then I looked at the python examples that come with NetworkManager. I’m not that great at python, but I fudged my way through them. Many of the examples did not run, and  I suspected that the version of NetworkManager I had was different to the one the examples were aimed at. Anyway, the biggest set of hints was in the update-secrets.py example, and eventually I hacked it and came up with the following python script (change-dns.py … Note that you might need to tailor this script if your settings is not Settings/0)

#!/usr/bin/env python
import dbus
import sys
import socket,struct

def dottedQuadToNum(ip):
   "convert decimal dotted quad string to long integer"
   return struct.unpack('<L',socket.inet_aton(ip))[0]

if len(sys.argv) != 2:
   sys.exit("Must supply ip address for dns; eg. 192.168.0.1")

x = dottedQuadToNum(sys.argv[1])
bus = dbus.SystemBus()
proxy = bus.get_object("org.freedesktop.NetworkManager", "/org/freedesktop/NetworkManager/Settings/0")

settings = dbus.Interface(proxy, "org.freedesktop.NetworkManager.Settings.Connection")

config = settings.GetSettings()
s_ipv4 = config['ipv4']
s_ipv4['dns'] = dbus.Array([dbus.UInt32(x)], signature=dbus.Signature('u'), variant_level=1)

settings.Update(config)

So now I could run my script (called change-dns.py) with

./change-dns.py 192.168.0.123

And then run that GetSettings dbus-send again to check that it changed.

dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager \
/org/freedesktop/NetworkManager/Settings/0 \
org.freedesktop.NetworkManager.Settings.Connection.GetSettings

So yes it did work, but it does not immediately update /etc/resolv.conf, so I still manually updated /etc/resolv.conf. Checking on a test system, NetworkManager does overwrite /etc/resolv.conf with the correct value after you reboot.

What about if you want to set two nameservers and perhaps a default search (eg. ‘search local’). Try changing the end of the script to accept two IP addresses;

if len(sys.argv) != 3:
   sys.exit("Must supply two ip addresses for dns; eg. 192.168.0.1 192.168.0.2")

x = dottedQuadToNum(sys.argv[1])
y = dottedQuadToNum(sys.argv[2])

bus = dbus.SystemBus()
proxy = bus.get_object("org.freedesktop.NetworkManager", "/org/freedesktop/NetworkManager/Settings/0")

settings = dbus.Interface(proxy, "org.freedesktop.NetworkManager.Settings.Connection")

config = settings.GetSettings()
s_ipv4 = config['ipv4']

s_ipv4['dns'] = dbus.Array([dbus.UInt32(x), dbus.UInt32(y)], signature=dbus.Signature('u'), variant_level=1)
s_ipv4['dns-search'] = dbus.Array([dbus.String(u'local')], signature=dbus.Signature('s'), variant_level=1)

settings.Update(config)

Just a side note in case you are trawling the net trying to figure out how you can change an IP address on the command line using dbus, basically the change-dns.py script just needs the ‘s_ipv4['dns'] = dbus.Array….’ lines changed to something like;

s_ipv4['addresses'] = dbus.Array([dbus.Array([dbus.UInt32(1962977472L), dbus.UInt32(24L), dbus.UInt32(16820416L)], signature=dbus.Signature(‘u’))], signature=dbus.Signature(‘au’), variant_level=1)

The tricky part is figuring out the numbers. There are three numbers (all ending in a capital ‘L’). The first is the IP, the 2nd the netmask, the 3rd the default route. In the example above the IP address is 192.168.0.117. I usually use a programmers calculator, convert that to hex; C0A80075, then reverse all the octets; 7500A8C0 and then convert that to decimal; 1962977472 (of course, its much easier if you use the dottedQuadToNum function in the python script)

I haven’t posted for a while about my Scientific Linux 6.0 setup on my Thinkpad R60. It all kind of goes well. However SL 6.0 is kind of old, 6.1 has been out for a while and 6.2 is on the horizon (UPDATE: Ooops, I see its now out!). So I thought I’d upgrade to 6.1. But before jumping into the simple update process I thought I’d split my root mirror as a precaution. In recent years I just keep having problems with Linux Distro updates. Its not that the update process fails, it’s usually some oddball behaviour post upgrade that I can’t live with. Anyway, I do my mdadm fail and stop to detach one half of my boot and root mirrors and do the update to 6.1.

It all seemed to go well. I reboot … and it starts to boot but X windows does not start. I look at the Xorg log and there’s some Segmentation Fault without much detail, but I see its trying to access some dri stuff before the fault, and I know this dri stuff doesn’t exist when you have nomodeset set. See, I can’t get X to run reliably under SL 6.0 on my Thinkpad R60 without turning KMS off (I get lots of stupid flicker). Anyway, I think maybe this new kernel has fixed the flicker problem so I get rid of ‘nomodeset’ and reboot, and sure enough X now starts and looks kind of OK.

However leaving it on for a while the flickering starts to occur again, and then I start noticing these ‘kernel:Uhhuh. NMI received for unknown reason’ messages on my terminal sessions periodically. The system seems to still be going OK, but I do some research and find this centos page. So it’s not sounding like a good upgrade at this point; I can’t start X unless Kernel Mode Setting is set, and when I have Kernel Mode Setting set I can’t look at the screen because it flickers badly. Crazy!. So in less than an hour I’ve had enough pain and reverted my md mirrors back to SL 6.0.

So I’ve had Scientific Linux 6.0 64 bit on my Thinkpad R60 for a while now, and for the most part it has been fine. But one thing that has really annoyed me is that everytime there is a kernel update, I notice that the laptop fan is stuck on (when the system is essentially idle). I always think “This can’t be right” and reboot back off my old kernel.

The thing is the only kernel I have now that has ‘good’ fan behaviour is 2.6.32-71.24.1 which is the one that went in when I first installed SL 6.0. Under that kernel , when the R60 is relatively idle, the fan will come on at 2500 to 2900rpm or so for say 60 secs, then be completely off for another 60 secs.  That’s kind of how I expect it to be when the system is idle (it would be nicer if the fan was off for longer periods … but sometimes off is better than never off)

A few yum updates later I also have 2.6.32-71.29.1 and the latest being 2.6.32-131.2.1. These two always run with the fan stuck on at 3300rpm or so (while the R60 is essentially idle). Very annoying. So it’s been interesting to read the various ‘linux power regression’ articles on Phoronix lately. The leading cause of the Recent Linux Kernel Power Problems article explains a key source of the problem related to some ASPM subsystem … and a boot option workaround; “pcie_aspm=force”

So I had a go at adding that option into the grub.conf for the 2.6.32-131.2.1 kernel and amazingly the fan behaviour is back to the “right” behaviour. The fan now comes on for about 60 secs at 2800rpm, then goes off for 60 secs and so on. Cool stuff. Many thanks to the Phoronix site for figuring this out.

Now, a key thing to remember with Redhat/SL kernels is that they contain many patches from more recent kernels ‘backported’ into an older kernel such as 2.6.32 … and hence that partly explains why this problem exists on the recent SL 6.0 kernels.

So, as per my previous post, I was searching for an upgrade path for my Thinkpad R60 virtualistation server running Debian Lenny. I’d tried upgrading to Squeeze a few times … and didn’t like it … always reverting back to Lenny.

So after trying a few different distros/virtualisation hypervisors (using a spare disk), I settled on installing Scientific Linux 6.0 64 bit. Key reasons are;

• It’s effectively Redhat 6.0, and Redhat ‘target’ stability rather than ‘latest and greatest’ which is what I’m after with a virtualisation box.
• By using KVM virtualisation on SL 6.0, it contains a version of libvirt that will let you disable the memory balloon (the one in Squeeze always has the memory balloon enabled). The memory balloon stuff is more annoying than useful on my little laptop server.
• It didn’t seem too annoying. Well that’s what I thought. Less annoying the Debian Squeeze I guess.

So I’d been trying out SL 6 on a spare disk … so now I thought I’d put it ‘properly’ on the two 500GB drives in the R60. I’d always run Lenny with a mirrored root and /boot so I wanted to do the same with SL 6.0. I’d already set split mirrors on the two 500GB drives. The original md devices I used on the first disk were md2 and md3. They currently had an old Debian Squeeze on them with a deliberately failed mirror. The other disk now had md20 and md30 which was the original Lenny install that I had ‘split off’ prior to my last Squeeze upgrade (md20 and md30 also being a degraded mirror). So I still wanted to keep md20 and md30 for now in case I wanted to go back to Lenny, and I would overwrite md2 and md3 with SL 6.0. I still wanted to put SL onto RAID1 mirror devices, because at some point I would kill md20 and md30 on the 2nd disk and properly mirror md2 and md3 again.

So I wanted to install SL 6.0 onto md2 and md3 which were both currently failed mirrors. Do you think it would let me do this? Nope. It seems the anaconda installer won’t let you do this … which is very very sad. To me it seems an obvious thing that you’d want to do. I found redhat bug 188314 which is basically a depressing read.

So, I dug out yet another spare disk (an old 30GB IDE) , removed the 2nd 500GB drive, repartitioned the 30GB drive to match the root and /boot partition sizes and then used mdadm to add in the extra partitions and eventually md2 and md3 were active RAID1 mirrors.

Then I tried to install again (lucky you can boot off a USB attached DVD drive on the R60). This time I got past the partitioning hiccup and had it installed. I used the SL 6.0 x86_64 LiveMiniCD … which doesn’t ask too many questions and gives you a reasonable (for 2011) size install. One thing on my R60 that I learnt from previous attempts, that at the inital GRUB screen press a key to get the boot menu, cursor down to ‘install’, press tab, edit the 2nd line, then add the word ‘nomodeset’ onto the end of the main kernel boot line. The ‘nomodeset’ turns off KMS and prevents the extremely annoying flickering I would previously get.

Once I had it installed, I then went about getting it sorted to work much the same as my Lenny setup. I’ll include some of my notes further down for reference, and just discuss some of the aspects of KVM setup right now.

So in order to get KVM set up right, I installed the main virtualisation stuff;

yum groupinstall Virtualization
yum groupinstall “Virtualization Client”
yum groupinstall “Virtualization Platform”

Then I had to edit /etc/libvirt/libvirtd.conf in order to get rid of the dependence on PolicyKit (or attempt to get rid of the dependence). Then I copied in the xml libvirt files from my Lenny install and started modifying them. I changed the kvm binary references to /usr/libexec/qemu-kvm and as I mentioned in my previous post, my Windows 7 32 bit VM always crashed on shutdown until I switched the cpu from i686 to x86_64 (as I had recently upgraded the CPU in my R60 from a 32 bit T2400 to a 64 bit T7200). The machine type in the xml files is also a puzzle. Lenny has a completely different set of them compared to Redhat/SL. Lenny had ‘pc’ as the main machine type, and on most of my attempted Squeeze upgrades, somehow the upgrade process seemed to think it was smart to change all my ‘pc’ machine references to ‘pc-0.12′. I remember noticing that this makes Windows think it has a new network card, new hard drive etc….

With Redhat/SL the QEMU/KVM machine types are pc, rhel6.0.0, rhel5.5.0, rhel5.4.0. It says ‘pc’ is the same as ‘rhel6.0.0′, so if I’d left my libvirt xml files all with ‘pc’ they’d be automatically upgraded … just like Squeeze did.  I think I currently have rhel5.4.0 as the choice for my Windows 7 VM for lack of a better choice.

Anyway, in the process of getting Windows 7 going again, I got the dreaded ‘Windows Actiation error’. Sure it now thought that not only had the network card changed, the hard drive and now it thinks it has a new CPU. I thought one of the ‘features’ of virtualisation is to have the ability to have a ‘constant’ hardware abstraction.

So I try to activate Windows 7 online. That fails, so I have to ‘ring MS$’ which is definitely the low point of the week. I type in a very long string of numbers on the phone keypad, only to be told that I needed to be put through to an operator. I read the same long series of numbers to a human, who then reads a similarly long series of numbers back to me … which finally does the trick.

I’m thinking at this point … that maybe I should listen to all the people that comment on my ESXi posts and ditch KVM and just switch to ESXi.;-)

But I am a glutton for punishment.

Anyway, it is all kind of working OK now. I added in the line to disable the memory balloon for my KVM guests, and also found there was some ksm daemons that also needed to be disabled, as they sporadically ate lots of CPU as well.

My notes;

# Get rid of the leftovers from doing the LiveCD install
chkconfig --del livesys
chkconfig --del livesys-late
# Turn SSH on
chkconfig --level 345 sshd on
service sshd start
# Turn NetworkManager off.
chkconfig NetworkManager off
service NetworkManager off
# Updated eth0 and br0 network configs

edit /etc/sysconfig/network-scripts/ifcfg-eth0 and ifcfg-br0
chkconfig --level 345 network on
service network start

# The LiveCD never seems to set the hostname
set HOSTNAME in /etc/sysconfig/network
Add our own hostname and IP to /etc/hosts
Comment out hiddenmenu in /boot/grub/menu.lst. Also add in my Lenny boot menu item
# Fix up some sudo rules
visudo
sudo yum upgrade

# Add in all the virtualistion stuff
yum groupinstall Virtualization
yum groupinstall "Virtualization Client"
yum groupinstall "Virtualization Platform"

# Turn off ksm
chkconfig ksm off
chkconfig ksmtuned off
service ksmtuned stop
service ksm stop
#LiveCD leaves firewall on, so need to do some updates to /etc/sysconfig/iptables and restart the iptables service

#I installed the development tools and some other stuff

yum groupinstall "Development tools"
yum install libX11-devel
yum install sg3_utils
yum install expect
yum install libXfont-devel

#downloaded EPEL release 6.5 rpm to get EPEL set up

#Compiled X11rdp and xrdp from svn and cvs respectively. Needed automake1.9 for  X11rdp. Downloaded a srpm for it and compiled by hand.
xrdp 0.6 has changed a bit. Still has the bug where it eats heaps of CPU though, so added the changes mentioned on this page;

http://sourceforge.net/projects/xrdp/forums/forum/389417/topic/3706601

#Quite a few deps to handle to do the xrdp compile. Can't remember all of them
yum install pam-devel
yum install libXfixes-devel

So, my main virtualisation machine here at home is a Thinkpad R60 running Debian Lenny, with KVM as the virtualisation engine. This has been my main ‘server’ host for over a year and a half now. I have quite a few VMs on it, but most are for testing out stuff, so there is only one or two VMs that are active constantly. One is a Debian Lenny guest running Asterisk for the most part. The other is a Windows 7 VM. I’ve been reasonably happy with this setup. Key positive points to me are;

• I can use the host as a regular linux box with a graphical desktop
• Performance is OK
• The Debian Lenny guest is idle most of the time, and is lucky to consume 2 or 3% of the host cpu.
• It all works with a 32 bit VT capable CPU such as the T2400 core duo in the R60

Negatives of this setup are;

• The T2400 is only 32 bit so no 64 bit guests.
• Physical host CPU use when the Windows 7 guest runs is high; 14% or so. I eventually tracked this down to the USB tablet driver. Once removed, host CPU use for the related kvm process is about 2 to 3% for an idle Windows 7.
• You cannot do easy snapshots and reverts like you can say with Vmware’s products.

So Debian Squeeze is the new stable ‘Debian’, so I have attempted to upgrade my system. I’ve actually done this now several times, first using some of the ‘pre stable’ releases of Squeeze, as well as the current stable Squeeze. Each time I have reverted back to Lenny. Key negatives of Squeeze are;

• Host CPU use seems higher for all guests for me.
• The extra CPU use by the USB tablet driver in a Windows guest still seems to be there. Makes a big difference if you remove it.
• The Windows 7 guest now seems to BSOD (ie. crash) when you shut it down (I’m just using a virtual IDE disk and a virtual RTL3129)
• The Laptop fan seems to stay on more often now.

So this has had me looking at alternatives. I am a big fan of Vmware ESXi, but because of the 32 bit CPU in the R60 I am limited to ESXi 3.5 on it. With that in mind, I did my research and discovered that you can put a T7200 core 2 duo CPU into my R60 which is a 64 bit VT capable CPU. So I ordered one off eaby and have had it running in it for a few days now.

So here’s a quick breakdown of what I’ve tried lately. Some of these I’ve only set up on my desktop core 2 duo system, but I intend to try them all on the R60

• Squeeze 64 bit. Much the same as the problems I’ve had with Squeeze mentioned above.
• ESXi 3.5u5. This works pretty well. On the R60, all the hardware works, except when you do a shutdown on the console, you get a purple screen of death. Reboots are OK though.
• ESXi 4.0 (I will try 4.1 shortly, but I had the 4.0 CD handy). That works pretty good too. The main negatives for ESXi on this box are; 1) no graphical console anymore, 2) one disk has to ‘generally’ be set aside as ESXi vm storage, 3) I have to use Windows to administer it, 4) I can’t read any of the Thinkpad thermal or fan information.
• Scientific Linux 6.0 64 bit. I tried the ‘LiveMiniCD’ which is pretty good. It installs an IceWM desktop. I’ve been running my Asterisk VM and my Windows 7 VM on it under kvm. In this case CPU use for both is low, but the Asterisk VM CPU use is still quite a bit higher than my Lenny setup. Also the Windows 7 VM crashes on shutdown just like under Squeeze. Scientific Linux 6 and Squeeze both use a very similar version of qemu-kvm. I might persists a bit of Scientific Linux. It doesn’t seem too bad (for a Redhat clone). One problem I had with SL 6.0 was a strange flicker on the X desktop every 10 secs. Changing the GRUB kernel args to include ‘nomodeset’ (ie. turn off KMS), cured this.
• Xenserver 5.6 FP1 . I haven’t tried it on the R60, just my desktop core 2 duo box. This I thought worked pretty well … and Citrix have done a lot to make it ‘feel’ just like ESXi for the most part, and the Windows admin tool seems to let you do most stuff that Vmware’s admin client let’s you do. I did have problems installing the guest tools into a Windows 7 guest though. This was a week or too back and I can’t quite remember what it was. But I was impressed by it in general, and I’d like to spend more time looking at it (or XCP below)
• XCP 1.0. This to me looked to be almost exactly the same as Xenserver 5.6 FP1. It had exactly the same problems re the guest tools in Windows 7.

So still undecided at the moment. I need to do a bit more research on the ESXi front. The R60 can only handle 3GB of RAM, and I think the newer ESXi’s have quite a large default memory footprint (but I think there is a way to reduce it a bit). If I could sort out the problem with the Windows 7 BSOD on shutdown then I might consider Scientific Linux. Lot’s of options.

UPDATE (2011.5.2). The Windows 7 BSOD I was getting when performing a shutdown on Scientific Linux or Squeeze was a IRQL_NOT_LESS_OR_EQUAL. I think I’ve found what causes this. All my VMs are created with i686 as the CPU type in the libvirt xml files (since my old T2400 chip was 32 bit only). Of course now I have a 64 bit capable chip (T7200) and the new versions of qemu-kvm don’t seem to like me specifying a i686 for my Windows 7 32 bit instance. If I change the xml file to use x86_64 instead of i686, I can now (finally) gracefully shut the Windows 7 32 bit VM down.;

<type arch=’x86_64′ …

Another useful thing to keep my CPU utilisation low is to disable the memory balloon (which I discovered is enabled by default in more recent libvirt’s). You just need to add this to the <devices> section of your libvirt xml file;

<memballoon model=’none’/>

Yes, I know Snow Leopard has been out for ages, and I did actually purchase the ‘real’ install DVD for it some time ago, but well ‘Leopard’ seemed to be working fine on my Macbook, and well I’m very cynical of these claims that ‘upgrading is easy and trouble free’. However, I bought a Hitachi 5K500.B 500GB drive recently as either an upgrade for the 160GB drive in the macbook or as a replacement for the original 80GB drive in my R60.

Given that I had a new (larger) drive. I thought I’d play with how well Time Machine and the Migration Assistant work. So as a test I setup a large mount point on a linux box and shared it out using Samba to the Macbook, then did the defaults command below to enable a samba share as a Time Machine destination:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

I also did the mucking around with hdiutil to create the necessary sparsebundle (if you google for ‘TMShowUnsupportedNetworkVolumes hdiutil’ you’ll find a heap of references). It takes a long time to do the first backup. I then set it to ‘manual’ mode so it’s not doing continual backups. Then I did one last Time Machine backup and shut the Macbook down (running Leopard).

Now I swapped in the 500GB drive. Put in the snow leopard DVD, held down ‘c’ and let it boot up off the DVD. Then formatted the drive, and installed Snow Leopard (I picked a different account name than the one I used on my Leopard install, as Migration Assistant will complain later on if you try to restore into the account you are currently logged in as). Once I had SL installed, I then did a ‘connect to server’ in finder to mount my Samba share, then you need to click on the sparsebundle that you see, and your time machine backup should now be mounted. Now you can run ‘Migration Assistant’. Tell it you want to restore from a ‘Time Machine’ backup, then you should see your time machine backup to choose, then you get some screen with a few tick boxes and you have to wait while it says ‘Calculating’. I just left everything ticked and let it chug away restoring (it takes some time). Eventually, I just restarted, logged in as the username I used to use and voila it all looked suspiciously like my old Leopard setup.

The first thing I wanted to make sure was working was Mail. I started it up and it gives you some messages saying it needs to re-import everything and away it goes. Now, I use Mail.app as my main mail client, and I have an IMAP server that it connects to. I have some anal security in place for the IMAP server, so when Mail was restarted after the re-import I didn’t let it connect properly to the IMAP server. I then started going through my Inbox to see whether it looked OK. Everything seemed to be there, but I had a weird problem when I clicked on attachments. They would never load. There’d be some delay and then a popup highlighting there was a permission problem. This led to much trawling through Apple discussion threads and trying various things, but I could not get attachments to display. I just received that error instead

So I switched back to the original 160GB drive for a while. Then I tried again, clean format of the 500GB drive, installed SL, then this time I upgraded to 10.6.2, then did the Migration Assistant restore. No luck. Still had the same permission problem.

So by this time I was getting frustrated with Snow Leopard, so I thought I’d try to just put Leopard on the 500GB drive. Rather than use something sane like Carbon Copy Cloner or similar I thought I’d see how good a full restore from ‘Time Machine’ would go onto the 500GB drive. So with the old 160GB drive containing leopard, I did another Time Machine backup to the Samba share, shut down, swapped in the 500GB drive and booted off the Leopard install DVD that came with the macbook. Booted off the DVD, formatted the 500GB drive, then chose the option to restore from backup. This is where I learnt that the Apple install DVDs cannot mount a Samba share (or any kind of smb based share). They can mount a afp share though. So I thought, I’ll install netatalk on my Debian Lenny server that was running Samba, and just get it to share out the same mount that Samba was sharing.

This is when I discovered that the Debian Lenny version of netatalk cannot deal with encrypted passwords. Guess what OSX will only connect to? Yep, it only connects to a netatalk server that can do encrypted passwords (there is actually a defaults ‘switch’ that will allow you to connect to a netatalk user ; defaults write com.apple.AppleShareClient afp_cleartext_allow -bool true, but alas I could not get that to work from the shell environment offered on the install DVD), so then I ended up recompiling netatalk on Debian Lenny to handle encrypted passwords, which goes somehting like this;

$ sudo apt-get install libdb-dev
$ sudo apt-get install cdbs
$ sudo apt-get install libcrack2-dev
$ sudo apt-get install dh-buildinfo

$ cd debtmp
$ mkdir debtmp
$ sudo apt-get source netatalk
$ sudo apt-get install libcups2-dev
$ sudo apt-get build-dep netatalk
$ cd netatalk-2.0.3
$ sudo DEB_BUILD_OPTIONS=openssl fakeroot debian/rules binary
$ sudo dpkg -i ../netatalk_2.0.3-11+lenny1_amd64.deb

sudo afppasswd -ac
sudo afppasswd myuser

You might need to tweak some of the other files under /etc/netatalk too.

OK, so now using the notes on mounting here; I could mount my Time Machine backup from the Leopard install DVD;

mkdir /Volumes/tmachine
mount -t afp afp://username:password@server.hostname/tmachine /Volumes/tmachine

So then I kicked off my restore. Maybe 5 or 10 mins later the restore had encountered an error. I honestly can’t remember what it was. The discussion threads I found when googling for the error hinted that it has something to do with using the wrong install DVD (ie. one that came with a different Mac). I was definitely using the DVD that came with my Macbook, but I thought since I had the Snow Leopard DVD as well, then surely a full restore of a Leopard backup using the Snow Leopard boot DVD would end up with a Leopard system. So I did that, mounted the afp share OK, then started the restore … which continued just fine. Odd.

So that chugged away, and eventually I had a Leopard system on a bigger drive. I opened Mail.app, and again it said it needed to reimport Mail (just like for Snow Leopard). It chugged away for a bit, and then I had my Inbox open and it looked intact. I then found some of the messages with attachments I had had trouble with before and tried to open them. Instead of getting a permission error, I received no message at all … and in fact nothing happened (ie. my attachment still did not display). Hmmmm. Odd. It’s about now that I somehow realised that by blocking access to my IMAP server, I was preventing it from grabbing the attachment. This is probably obvious, but I imagine that Mail.app downloads attachment when you first click on them, and then caches them in the ~/Library/Mail\ Downloads directory, so that normally you’d be viewing the cached copy. However, I know that Time Machine ignores most cache directories in order to save space … so after my ‘fresh’ install plus restore, the cache directory was empty. I guess I was able to confirm my theory quickly by simply allowing my connection to my IMAP server to work (I noticed after I did this that there is a lot of activity in the ‘Activity’ window regarding caching etc.)

So after all that hoopla about Mail attachments, I thought I’ll reinstall Snow Leopard again. So same deal as before; boot off the SL DVD, format the 500GB drive, let it install SL, get it booted, perhaps upgrade to 10.6.3 or whatever, then use the Migration Assistant to restore from my Time Machine backup. This time when I ran Mail.app I let it connect to my IMAP server, and attachments worked as expected. All good

Apart from my funny Mail.app issue (which was sort of my fault), I was quite impressed with the whole Time Machine backup/restore. I am often asked by family or friends about ‘backing up their Windows PC’, and well there are a bunch of ways to back things up … but I always think a backup is only as good as your ability to restore it. My thoughts were that I could probably talk someone through doing a full restore of a Mac using a Time Machine backup … over the phone (maybe not from my convoluted AFP/Samba shared Time Machine server, but from a local USB drive). I am sure there are Windows products that claim to do something similar, but an enticing thing is that Time Machine is pretty much supported ‘out of the box’.

My upgrade was not all good. I have several vmware fusion guests on my Macbook, and when these were restored on the 500GB drive, all of them seemed to have very scary filesystem errors (the ones you get when there’s been some serious corruption). I thought maybe this was to do with Time Machine, but I ended up using my crazy rsync backups to restore from as well … and I still had the same filesystem errors. In the end, I booted back up off the 160GB drive in the macbook, and went through every Vmware guest and told them to power down (pretty much all of them were just ‘suspended’), then I did my rsync backup again, and restored them with the 500GB drive inside. And now they all work and have no filesystem errors. Obviously there is some strange inconsistency that occurs with the guest ‘state’ information during the upgrade to Snow Leopard.

One thing that would be nice to have in linux LVM is the ability to take a snapshot of a logical volume, make some changes then ‘roll back’ to the state preserved in the snapshot. If you look at the current set of lv commands on most linux distros there is no such option to ‘revert to snapshot’. Now, that I’m using KVM more, the ability to ‘rollback’ a virtual machine installed on a logical volume would be particularly handy.

Well it turns out you can ‘revert to snapshot’, but you just have to use a non-standard tool and some dmsetup commands. The tool is called dm-merge. Go to the dm-merge git repository or download the latest dm-merge snapshot . dm-merge just looks at the ‘changed blocks’ and pushes the old copies of them back into the original logical volume. You’ll need to compile the dm-merge program (just do a ‘make’). dm-merge comes with some doco. It explains two possible methods; reverting with an active logical volume OR reverting with an inactive one. In the former case you’re attempting to revert to a snapshot when the filesystem you’re trying to change is already mounted. I imagine that to be a very dangerous activity (but I guess it works well enough if there is little filesystem activity). I chose the latter (safer) method.

Here are my notes on the procedure. If this were a document for work, I would a) tell people they were crazy to even attempt this and b) if you’re still crazy make sure you make a backup. At least try this procedure out on a test machine first. The possibility of screwing your disk by virtue of a small typing mistake is quite high.

In the example below, I had a logical volume /dev/vgz/lvubuntu910full32bit that I was using as my hda drive in a KVM guest machine, and I had created a snapshot ‘snap1′ beforehand;

   lvcreate --size 1G --snapshot --name snap1 /dev/vgz/lvubuntu910full32bit

One important thing I realised recently is that the amount of space you allocate to a snapshot (eg. 1GB in the example above) can actually be resized without killing the snapshot (eg. lvresize -L 20G /dev/vgz/snap1)

Most of the following is a rehash of what is in the README that comes with dm-merge. I’m only referring to the method involving an inactive logical volume

# Firstly, shut down access to your logical volume (ie. /dev/vgz/vgzlvubuntu910full32bit).
   Since this is a KVM guest, all I did was shut down the guest. If you're doing this on a
   mounted logical volume then you should umount it.

# Duplicate the tables of the logical volumes and snapshot storage (note the use of the
   special -real and -cow suffixes)

dmsetup table vgz-lvubuntu910full32bit-real |dmsetup create tmplv
dmsetup table vgz-snap1-cow |dmsetup create tmpcow

# Deactivate the main lv

lvchange -a n /dev/vgz/lvubuntu910full32bit

# flush buffers

blockdev --flushbufs /dev/mapper/{tmplv,tmpcow}

# Do a dm-merge test run. You see lots of lines that start with 'dd' and a summary at the bottom.

dm-merge -i /dev/mapper/tmpcow -o /dev/mapper/tmplv -vd
...
dd of="/dev/mapper/tmplv" seek=2135444 if='/dev/mapper/tmpcow' iflag=direct skip=8275 count=1 bs=8b
dd of="/dev/mapper/tmplv" seek=2135445 if='/dev/mapper/tmpcow' iflag=direct skip=8276 count=1 bs=8b
dd of="/dev/mapper/tmplv" seek=2135446 if='/dev/mapper/tmpcow' iflag=direct skip=8277 count=1 bs=8b
dd of="/dev/mapper/tmplv" seek=5056466 if='/dev/mapper/tmpcow' iflag=direct skip=8278 count=1 bs=8b
dd of="/dev/mapper/tmplv" seek=5056467 if='/dev/mapper/tmpcow' iflag=direct skip=8279 count=1 bs=8b
Found 8246 exceptions of chunksize 4096, total size 33775616 bytes (32984 KiB, 32.211 MiB, 0.031 GiB).

# Do it for real. You get a couple of lines of output plus the same summary line.

 dm-merge -i /dev/mapper/tmpcow -o /dev/mapper/tmplv -f

Artificial sleep (1 second)
Found a proper MAGIC header: 0x70416e53
valid = 1
version = 1
chunk_size = 8 (4096 bytes)
Found 8246 exceptions of chunksize 4096, total size 33775616 bytes (32984 KiB, 32.211 MiB, 0.031 GiB).

# Flush buffers again

blockdev --flushbufs /dev/mapper/{tmplv,tmpcow}

# Remove the tmp lv's. These are just references to data, so it won't delete your real data.

dmsetup remove tmplv
dmsetup remove tmpcow

# Activate it

lvchange -a y /dev/vgz/lvubuntu910full32bit

Now try to start your KVM guest again … or remount it to check it out and confirm that it’s reverted to the original state. If you snapshotted a live KVM guest or a mounted filesystem, then you may find that your KVM guest will do a filesystem check on boot (fair enough), or you may need to run fsck on your filesystem before you can mount it.

Don’t forget to remove the snapshot you initially made (eg. lvremove /dev/vgz/snap1)

One thing to note is the ‘revert to snapshot’ capability has been ‘coming’ for a while in mainstream LVM. There’s a recent mention on LWN about it

For some time now, my old Thinkpad T42 has been running as a lightweight server at home. Some people think I’m a bit mad using a laptop as a server, but my vague reasons are a) they’re generally quiet, b) they don’t use a heap of power, c) the battery is in some ways a built-in UPS and d) whenever I’ve tried to take a regular desktop system and make it quiet I end up spending lots of $$$ on various quiet fans and power supplies only to be moderately satisfied with the results. To me, an old 2nd hand laptop is generally better value. Of course, you can’t fit many disks into laptops, but with the T42 I discovered that even though the primary drive has to be a 2.5″ IDE drive, you can fit a large 2.5″ sata drive into them by buying a sata HDD caddy (look on ebay).

Anyway, I have Debian Lenny on the T42 with several OpenVZ containers for things like Asterisk and a mercurial server. In addition it’s a streaming server, as I have two internal drives plus a large external 3.5″ USB drive.

The T42 is getting old, and I’d like to be able to run KVM on it … because basically I like KVM, but KVM needs a VT capable CPU. So I kept my eye out for a cheap 2nd hand VT capable laptop. I ended up buying a 2nd hand Thinkpad R60. This is probably one of the ugliest Thinkpads ever made which might explain the good price I got it for. I certainly wouldn’t buy one to lug around, but sitting on a desk at home quietly running a number of VMs is all I really wanted. It has a T2400 core duo (not core 2) cpu. Technically it is ‘VT capable’, but its a bit of an odd one (which probably applies to the T2500, T2600 etc) in that its not a 64 bit CPU like most other VT cpus. Other than being ugly and having an odd CPU, this R60 does have a lot going for it; 3GB of RAM, DVD burner, firewire, wifi, bluetooth, cardbus and expresscard slots (so I could add an esata expresscard device to add some highspeed desktop drives later on). And the HDD caddies I had for my T42 also fit in the R60.

I originally tried Ubuntu 9.10 on the R60, but the version of qemu they use in combination with KVM actually does not work properly on this old T2400 CPU. I could seemingly install a guest OS, but after reboot I had errors galore. Something to do with the ‘No execute’ bit I think. Anyway, this was yet another case of ubuntu quickly irritating me, so I ended up installing something else. Back to Debian Lenny. And of course VM’s work fine under Lenny and it’s older KVM and older qemu tools.

So far I’ve just run debian lenny 32 bit and Windows 2008 R2 32 bit in some KVM VMs. Both seem to run quite well. The laptop has 3GB of ram, so I can run quite a few VMs at the same time OK. I’ve migrated my asterisk installation from the openVZ container it was in on my T42 to a KVM VM on the R60. It works quite well.

One thing that does bug me about using KVM is the lack of easy to use snapshots. ESXi spoils you in the simplicity of it’s snapshots on running or shutdown VMs. You can use them for backups or easily roll back to them. However, KVM has a haphazard collection of tools that don’t quite do the same thing. virt-manager offers no option for snapshots. If you use virsh instead it does have a ‘save’ option to snapshot a machine … but it seems to always shut the machine down when it runs. If you don’t use libvirtd to manage your VMs and instead use kvm directly, then you can interact directly with the qemu monitor of a VM and it has a savevm option which does something akin to a snapshot, but everytime I’ve tried it on a live VM, it pauses the VM for several minutes which is somewhat annoying. The closest thing to ESXi snapshots that I can think of is to use logical volumes for the VM’s virtual disk, then use LVM snapshots to create snapshots of them. These aren’t full ‘state’ snapshots that include a ram snapshot etc, but are probably good enough for my purposes. The tricky part is rolling back to a snapshot which is not really supported with logical volumes out of the box, but you can do it with something like the dm-merge tool.

I also tried Fedora 12 on the R60. The motivation was that the version of KVM on Debian Lenny is old, and Fedora 12 has quite a number of new features of KVM including better qcow2 performance and something that looks for identical memory pages in VMs to reduce memory usage. This latter feature appears as some ksmd daemon that seems to hog 50% of one CPU. Very annoying, so I killed it. I was hoping that the qcow2 enhancements might solve my ‘snapshots’ issue, but so far it doesn’t look like it. I need to investigate more. In the meantime, I’ll go back to Debian Lenny on the R60.

Of course, I did try running ESXi on the R60. ESXi 3.5u4 seemingly worked perfectly on it. I didn’t't need any fancy DIY network drivers. It just picked up the internal broadcom gigabit and was able to see the two internal drives. The only problem is that I get a Purple screen of death when I try do a shutdown (some error to do with ACPI, and setting acpi=off in the kernel options does not help). However, if you tell ESXi to reboot it reboots as expected (UPDATE: I have just tried ESXi 3.5u5 … and is has the same purple screen when you attempt a shutdown). I didn’t bother trying ESXi 4, as I am pretty sure that needs a 64 bit CPU.