Which leads me to this post. I was helping to set up a Fedora 16 system recently that had already been installed … and given that NetworkManager is the default, the sysadmin who set it up left it using NetworkManager. The thing is they had left out the DNS nameserver settings and I needed some name resolution. Of course the easy solution is edit /etc/resolv.conf and add in a nameserver line and in less than 30 seconds (assuming you have the network connectivity in place) you have name resolution. This is what I did and of course it worked and I was happy.

But then there was a power failure at the site and the server rebooted. It came up OK and the first thing I noticed was that /etc/resolv.conf did not have a nameserver entry in it. I knew straight away “Oh, that’s NetworkManager overwriting /etc/resolv.conf on boot”. So what to do to fix it? The normal answer is “Hop on the console, go to whatever Network preferences is called this week and add in the DNS IP to the Wired Connection in NetworkManager”. Unfortunately I did not have console access to this server. I did have a remote ssh connection so I tried logging in with x forwarding on, checked that I could launch an xterm, and tried running nm-applet .. which just hung and did not launch the GUI. I also sudo’d to root, made sure all my xauth and DISPLAY was right, checked that I could launch an xterm, then tried nm-applet … and again ‘nothing’. The thing is I had kind of seem this behaviour before on redhat-ish systems … and come to the conclusion that “PolicyKit is evil”.

So what next? Surely there must be a command line interface to NetworkManager? Nope. The closest thing is nmcli, but it has the great non-feature of not letting you change much. Sure it can give you some great status info, let you enable and disable some stuff, but no-can-do if you want to just update the DNS server.

Some googling later, and I’m finding some info about using dbus-send to query and control NetworkManager. In the examples below each dbus-send command gives you a little bit more info for typing in the next command (eg. the first one returns /org/freedesktop/NetworkManager/ActiveConnection/0 on my system for the list of active connections)

#work out the active connections
 dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager \
/org/freedesktop/NetworkManager org.freedesktop.DBus.Properties.Get \
string:org.freedesktop.NetworkManager string:ActiveConnections

#get the list of devices
 dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager \
 /org/freedesktop/NetworkManager/ActiveConnection/0 \
org.freedesktop.DBus.Properties.Get \
string:org.freedesktop.NetworkManager.Connection.Active string:Devices

#get the list of IP4Config stuff
 dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager \
/org/freedesktop/NetworkManager/Devices/1 \
org.freedesktop.DBus.Properties.Get \
string:org.freedesktop.NetworkManager.Device string:Ip4Config

# print the nameservers. I knew this was the right command as I
#tried it on a test system that already had a DNS server assigned
 dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager \
/org/freedesktop/NetworkManager/IP4Config/0 \
org.freedesktop.DBus.Properties.Get \
string:org.freedesktop.NetworkManager.IP4Config string:Nameservers

OK now, so I could drill down to get the nameserver detail. Great. Surely there is some easy way to then set it? After more googling it eventually transpired that the IP4Config stuff I was getting was a read-only dbussy thing. And that there must be some other way to actually ‘write’ to the settings. After more sleuthing (and using the GUI tool d-feet), I worked out this command;

dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager \
/org/freedesktop/NetworkManager/Settings/0 \
org.freedesktop.NetworkManager.Settings.Connection.GetSettings

That prints out a heap of stuff including my IP address and DNS info, but its all in UINT32 type format, so an IP address like 192.168.0.100 is shown as;

uint32 1677764800

So how did I figure out that ‘Settings/0′ bit? You can get a list of the settings available with;

dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager \
/org/freedesktop/NetworkManager/Settings \
 org.freedesktop.NetworkManager.Settings.ListConnections

OK, now I thought there must be a way of using dbus-send to update the DNS address. I tried a few different ideas that all failed, and then found some stackoverflow comment indicating in the dbus-send man page how ‘D-Bus supports more types than these, but dbus-send currently does not’. Great.

So then I looked at the python examples that come with NetworkManager. I’m not that great at python, but I fudged my way through them. Many of the examples did not run, and  I suspected that the version of NetworkManager I had was different to the one the examples were aimed at. Anyway, the biggest set of hints was in the update-secrets.py example, and eventually I hacked it and came up with the following python script (change-dns.py … Note that you might need to tailor this script if your settings is not Settings/0)

#!/usr/bin/env python
import dbus
import sys
import socket,struct

def dottedQuadToNum(ip):
   "convert decimal dotted quad string to long integer"
   return struct.unpack('<L',socket.inet_aton(ip))[0]

if len(sys.argv) != 2:
   sys.exit("Must supply ip address for dns; eg. 192.168.0.1")

x = dottedQuadToNum(sys.argv[1])
bus = dbus.SystemBus()
proxy = bus.get_object("org.freedesktop.NetworkManager", "/org/freedesktop/NetworkManager/Settings/0")

settings = dbus.Interface(proxy, "org.freedesktop.NetworkManager.Settings.Connection")

config = settings.GetSettings()
s_ipv4 = config['ipv4']
s_ipv4['dns'] = dbus.Array([dbus.UInt32(x)], signature=dbus.Signature('u'), variant_level=1)

settings.Update(config)

So now I could run my script (called change-dns.py) with

./change-dns.py 192.168.0.123

And then run that GetSettings dbus-send again to check that it changed.

dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager \
/org/freedesktop/NetworkManager/Settings/0 \
org.freedesktop.NetworkManager.Settings.Connection.GetSettings

So yes it did work, but it does not immediately update /etc/resolv.conf, so I still manually updated /etc/resolv.conf. Checking on a test system, NetworkManager does overwrite /etc/resolv.conf with the correct value after you reboot.

What about if you want to set two nameservers and perhaps a default search (eg. ‘search local’). Try changing the end of the script to accept two IP addresses;

if len(sys.argv) != 3:
   sys.exit("Must supply two ip addresses for dns; eg. 192.168.0.1 192.168.0.2")

x = dottedQuadToNum(sys.argv[1])
y = dottedQuadToNum(sys.argv[2])

bus = dbus.SystemBus()
proxy = bus.get_object("org.freedesktop.NetworkManager", "/org/freedesktop/NetworkManager/Settings/0")

settings = dbus.Interface(proxy, "org.freedesktop.NetworkManager.Settings.Connection")

config = settings.GetSettings()
s_ipv4 = config['ipv4']

s_ipv4['dns'] = dbus.Array([dbus.UInt32(x), dbus.UInt32(y)], signature=dbus.Signature('u'), variant_level=1)
s_ipv4['dns-search'] = dbus.Array([dbus.String(u'local')], signature=dbus.Signature('s'), variant_level=1)

settings.Update(config)

Just a side note in case you are trawling the net trying to figure out how you can change an IP address on the command line using dbus, basically the change-dns.py script just needs the ‘s_ipv4['dns'] = dbus.Array….’ lines changed to something like;

s_ipv4['addresses'] = dbus.Array([dbus.Array([dbus.UInt32(1962977472L), dbus.UInt32(24L), dbus.UInt32(16820416L)], signature=dbus.Signature(‘u’))], signature=dbus.Signature(‘au’), variant_level=1)

The tricky part is figuring out the numbers. There are three numbers (all ending in a capital ‘L’). The first is the IP, the 2nd the netmask, the 3rd the default route. In the example above the IP address is 192.168.0.117. I usually use a programmers calculator, convert that to hex; C0A80075, then reverse all the octets; 7500A8C0 and then convert that to decimal; 1962977472 (of course, its much easier if you use the dottedQuadToNum function in the python script)

It all seemed to go well. I reboot … and it starts to boot but X windows does not start. I look at the Xorg log and there’s some Segmentation Fault without much detail, but I see its trying to access some dri stuff before the fault, and I know this dri stuff doesn’t exist when you have nomodeset set. See, I can’t get X to run reliably under SL 6.0 on my Thinkpad R60 without turning KMS off (I get lots of stupid flicker). Anyway, I think maybe this new kernel has fixed the flicker problem so I get rid of ‘nomodeset’ and reboot, and sure enough X now starts and looks kind of OK.

However leaving it on for a while the flickering starts to occur again, and then I start noticing these ‘kernel:Uhhuh. NMI received for unknown reason’ messages on my terminal sessions periodically. The system seems to still be going OK, but I do some research and find this centos page. So it’s not sounding like a good upgrade at this point; I can’t start X unless Kernel Mode Setting is set, and when I have Kernel Mode Setting set I can’t look at the screen because it flickers badly. Crazy!. So in less than an hour I’ve had enough pain and reverted my md mirrors back to SL 6.0.

The thing is the only kernel I have now that has ‘good’ fan behaviour is 2.6.32-71.24.1 which is the one that went in when I first installed SL 6.0. Under that kernel , when the R60 is relatively idle, the fan will come on at 2500 to 2900rpm or so for say 60 secs, then be completely off for another 60 secs.  That’s kind of how I expect it to be when the system is idle (it would be nicer if the fan was off for longer periods … but sometimes off is better than never off)

A few yum updates later I also have 2.6.32-71.29.1 and the latest being 2.6.32-131.2.1. These two always run with the fan stuck on at 3300rpm or so (while the R60 is essentially idle). Very annoying. So it’s been interesting to read the various ‘linux power regression’ articles on Phoronix lately. The leading cause of the Recent Linux Kernel Power Problems article explains a key source of the problem related to some ASPM subsystem … and a boot option workaround; “pcie_aspm=force”

So I had a go at adding that option into the grub.conf for the 2.6.32-131.2.1 kernel and amazingly the fan behaviour is back to the “right” behaviour. The fan now comes on for about 60 secs at 2800rpm, then goes off for 60 secs and so on. Cool stuff. Many thanks to the Phoronix site for figuring this out.

Now, a key thing to remember with Redhat/SL kernels is that they contain many patches from more recent kernels ‘backported’ into an older kernel such as 2.6.32 … and hence that partly explains why this problem exists on the recent SL 6.0 kernels.

So after trying a few different distros/virtualisation hypervisors (using a spare disk), I settled on installing Scientific Linux 6.0 64 bit. Key reasons are;

• It’s effectively Redhat 6.0, and Redhat ‘target’ stability rather than ‘latest and greatest’ which is what I’m after with a virtualisation box.
• By using KVM virtualisation on SL 6.0, it contains a version of libvirt that will let you disable the memory balloon (the one in Squeeze always has the memory balloon enabled). The memory balloon stuff is more annoying than useful on my little laptop server.
• It didn’t seem too annoying. Well that’s what I thought. Less annoying the Debian Squeeze I guess.

So I’d been trying out SL 6 on a spare disk … so now I thought I’d put it ‘properly’ on the two 500GB drives in the R60. I’d always run Lenny with a mirrored root and /boot so I wanted to do the same with SL 6.0. I’d already set split mirrors on the two 500GB drives. The original md devices I used on the first disk were md2 and md3. They currently had an old Debian Squeeze on them with a deliberately failed mirror. The other disk now had md20 and md30 which was the original Lenny install that I had ‘split off’ prior to my last Squeeze upgrade (md20 and md30 also being a degraded mirror). So I still wanted to keep md20 and md30 for now in case I wanted to go back to Lenny, and I would overwrite md2 and md3 with SL 6.0. I still wanted to put SL onto RAID1 mirror devices, because at some point I would kill md20 and md30 on the 2nd disk and properly mirror md2 and md3 again.

So I wanted to install SL 6.0 onto md2 and md3 which were both currently failed mirrors. Do you think it would let me do this? Nope. It seems the anaconda installer won’t let you do this … which is very very sad. To me it seems an obvious thing that you’d want to do. I found redhat bug 188314 which is basically a depressing read.

So, I dug out yet another spare disk (an old 30GB IDE) , removed the 2nd 500GB drive, repartitioned the 30GB drive to match the root and /boot partition sizes and then used mdadm to add in the extra partitions and eventually md2 and md3 were active RAID1 mirrors.

Then I tried to install again (lucky you can boot off a USB attached DVD drive on the R60). This time I got past the partitioning hiccup and had it installed. I used the SL 6.0 x86_64 LiveMiniCD … which doesn’t ask too many questions and gives you a reasonable (for 2011) size install. One thing on my R60 that I learnt from previous attempts, that at the inital GRUB screen press a key to get the boot menu, cursor down to ‘install’, press tab, edit the 2nd line, then add the word ‘nomodeset’ onto the end of the main kernel boot line. The ‘nomodeset’ turns off KMS and prevents the extremely annoying flickering I would previously get.

Once I had it installed, I then went about getting it sorted to work much the same as my Lenny setup. I’ll include some of my notes further down for reference, and just discuss some of the aspects of KVM setup right now.

So in order to get KVM set up right, I installed the main virtualisation stuff;

yum groupinstall Virtualization
yum groupinstall “Virtualization Client”
yum groupinstall “Virtualization Platform”

Then I had to edit /etc/libvirt/libvirtd.conf in order to get rid of the dependence on PolicyKit (or attempt to get rid of the dependence). Then I copied in the xml libvirt files from my Lenny install and started modifying them. I changed the kvm binary references to /usr/libexec/qemu-kvm and as I mentioned in my previous post, my Windows 7 32 bit VM always crashed on shutdown until I switched the cpu from i686 to x86_64 (as I had recently upgraded the CPU in my R60 from a 32 bit T2400 to a 64 bit T7200). The machine type in the xml files is also a puzzle. Lenny has a completely different set of them compared to Redhat/SL. Lenny had ‘pc’ as the main machine type, and on most of my attempted Squeeze upgrades, somehow the upgrade process seemed to think it was smart to change all my ‘pc’ machine references to ‘pc-0.12′. I remember noticing that this makes Windows think it has a new network card, new hard drive etc….

With Redhat/SL the QEMU/KVM machine types are pc, rhel6.0.0, rhel5.5.0, rhel5.4.0. It says ‘pc’ is the same as ‘rhel6.0.0′, so if I’d left my libvirt xml files all with ‘pc’ they’d be automatically upgraded … just like Squeeze did.  I think I currently have rhel5.4.0 as the choice for my Windows 7 VM for lack of a better choice.

Anyway, in the process of getting Windows 7 going again, I got the dreaded ‘Windows Actiation error’. Sure it now thought that not only had the network card changed, the hard drive and now it thinks it has a new CPU. I thought one of the ‘features’ of virtualisation is to have the ability to have a ‘constant’ hardware abstraction.

So I try to activate Windows 7 online. That fails, so I have to ‘ring MS$’ which is definitely the low point of the week. I type in a very long string of numbers on the phone keypad, only to be told that I needed to be put through to an operator. I read the same long series of numbers to a human, who then reads a similarly long series of numbers back to me … which finally does the trick.

I’m thinking at this point … that maybe I should listen to all the people that comment on my ESXi posts and ditch KVM and just switch to ESXi.;-)

But I am a glutton for punishment.

Anyway, it is all kind of working OK now. I added in the line to disable the memory balloon for my KVM guests, and also found there was some ksm daemons that also needed to be disabled, as they sporadically ate lots of CPU as well.

My notes;

# Get rid of the leftovers from doing the LiveCD install
chkconfig --del livesys
chkconfig --del livesys-late
# Turn SSH on
chkconfig --level 345 sshd on
service sshd start
# Turn NetworkManager off.
chkconfig NetworkManager off
service NetworkManager off
# Updated eth0 and br0 network configs

edit /etc/sysconfig/network-scripts/ifcfg-eth0 and ifcfg-br0
chkconfig --level 345 network on
service network start

# The LiveCD never seems to set the hostname
set HOSTNAME in /etc/sysconfig/network
Add our own hostname and IP to /etc/hosts
Comment out hiddenmenu in /boot/grub/menu.lst. Also add in my Lenny boot menu item
# Fix up some sudo rules
visudo
sudo yum upgrade

# Add in all the virtualistion stuff
yum groupinstall Virtualization
yum groupinstall "Virtualization Client"
yum groupinstall "Virtualization Platform"

# Turn off ksm
chkconfig ksm off
chkconfig ksmtuned off
service ksmtuned stop
service ksm stop
#LiveCD leaves firewall on, so need to do some updates to /etc/sysconfig/iptables and restart the iptables service

#I installed the development tools and some other stuff

yum groupinstall "Development tools"
yum install libX11-devel
yum install sg3_utils
yum install expect
yum install libXfont-devel

#downloaded EPEL release 6.5 rpm to get EPEL set up

#Compiled X11rdp and xrdp from svn and cvs respectively. Needed automake1.9 for  X11rdp. Downloaded a srpm for it and compiled by hand.
xrdp 0.6 has changed a bit. Still has the bug where it eats heaps of CPU though, so added the changes mentioned on this page;

http://sourceforge.net/projects/xrdp/forums/forum/389417/topic/3706601

#Quite a few deps to handle to do the xrdp compile. Can't remember all of them
yum install pam-devel
yum install libXfixes-devel

• I can use the host as a regular linux box with a graphical desktop
• Performance is OK
• The Debian Lenny guest is idle most of the time, and is lucky to consume 2 or 3% of the host cpu.
• It all works with a 32 bit VT capable CPU such as the T2400 core duo in the R60

Negatives of this setup are;

• The T2400 is only 32 bit so no 64 bit guests.
• Physical host CPU use when the Windows 7 guest runs is high; 14% or so. I eventually tracked this down to the USB tablet driver. Once removed, host CPU use for the related kvm process is about 2 to 3% for an idle Windows 7.
• You cannot do easy snapshots and reverts like you can say with Vmware’s products.

So Debian Squeeze is the new stable ‘Debian’, so I have attempted to upgrade my system. I’ve actually done this now several times, first using some of the ‘pre stable’ releases of Squeeze, as well as the current stable Squeeze. Each time I have reverted back to Lenny. Key negatives of Squeeze are;

• Host CPU use seems higher for all guests for me.
• The extra CPU use by the USB tablet driver in a Windows guest still seems to be there. Makes a big difference if you remove it.
• The Windows 7 guest now seems to BSOD (ie. crash) when you shut it down (I’m just using a virtual IDE disk and a virtual RTL3129)
• The Laptop fan seems to stay on more often now.

So this has had me looking at alternatives. I am a big fan of Vmware ESXi, but because of the 32 bit CPU in the R60 I am limited to ESXi 3.5 on it. With that in mind, I did my research and discovered that you can put a T7200 core 2 duo CPU into my R60 which is a 64 bit VT capable CPU. So I ordered one off eaby and have had it running in it for a few days now.

So here’s a quick breakdown of what I’ve tried lately. Some of these I’ve only set up on my desktop core 2 duo system, but I intend to try them all on the R60

• Squeeze 64 bit. Much the same as the problems I’ve had with Squeeze mentioned above.
• ESXi 3.5u5. This works pretty well. On the R60, all the hardware works, except when you do a shutdown on the console, you get a purple screen of death. Reboots are OK though.
• ESXi 4.0 (I will try 4.1 shortly, but I had the 4.0 CD handy). That works pretty good too. The main negatives for ESXi on this box are; 1) no graphical console anymore, 2) one disk has to ‘generally’ be set aside as ESXi vm storage, 3) I have to use Windows to administer it, 4) I can’t read any of the Thinkpad thermal or fan information.
• Scientific Linux 6.0 64 bit. I tried the ‘LiveMiniCD’ which is pretty good. It installs an IceWM desktop. I’ve been running my Asterisk VM and my Windows 7 VM on it under kvm. In this case CPU use for both is low, but the Asterisk VM CPU use is still quite a bit higher than my Lenny setup. Also the Windows 7 VM crashes on shutdown just like under Squeeze. Scientific Linux 6 and Squeeze both use a very similar version of qemu-kvm. I might persists a bit of Scientific Linux. It doesn’t seem too bad (for a Redhat clone). One problem I had with SL 6.0 was a strange flicker on the X desktop every 10 secs. Changing the GRUB kernel args to include ‘nomodeset’ (ie. turn off KMS), cured this.
• Xenserver 5.6 FP1 . I haven’t tried it on the R60, just my desktop core 2 duo box. This I thought worked pretty well … and Citrix have done a lot to make it ‘feel’ just like ESXi for the most part, and the Windows admin tool seems to let you do most stuff that Vmware’s admin client let’s you do. I did have problems installing the guest tools into a Windows 7 guest though. This was a week or too back and I can’t quite remember what it was. But I was impressed by it in general, and I’d like to spend more time looking at it (or XCP below)
• XCP 1.0. This to me looked to be almost exactly the same as Xenserver 5.6 FP1. It had exactly the same problems re the guest tools in Windows 7.

So still undecided at the moment. I need to do a bit more research on the ESXi front. The R60 can only handle 3GB of RAM, and I think the newer ESXi’s have quite a large default memory footprint (but I think there is a way to reduce it a bit). If I could sort out the problem with the Windows 7 BSOD on shutdown then I might consider Scientific Linux. Lot’s of options.

UPDATE (2011.5.2). The Windows 7 BSOD I was getting when performing a shutdown on Scientific Linux or Squeeze was a IRQL_NOT_LESS_OR_EQUAL. I think I’ve found what causes this. All my VMs are created with i686 as the CPU type in the libvirt xml files (since my old T2400 chip was 32 bit only). Of course now I have a 64 bit capable chip (T7200) and the new versions of qemu-kvm don’t seem to like me specifying a i686 for my Windows 7 32 bit instance. If I change the xml file to use x86_64 instead of i686, I can now (finally) gracefully shut the Windows 7 32 bit VM down.;

<type arch=’x86_64′ …

Another useful thing to keep my CPU utilisation low is to disable the memory balloon (which I discovered is enabled by default in more recent libvirt’s). You just need to add this to the <devices> section of your libvirt xml file;

<memballoon model=’none’/>

Given that I had a new (larger) drive. I thought I’d play with how well Time Machine and the Migration Assistant work. So as a test I setup a large mount point on a linux box and shared it out using Samba to the Macbook, then did the defaults command below to enable a samba share as a Time Machine destination:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

I also did the mucking around with hdiutil to create the necessary sparsebundle (if you google for ‘TMShowUnsupportedNetworkVolumes hdiutil’ you’ll find a heap of references). It takes a long time to do the first backup. I then set it to ‘manual’ mode so it’s not doing continual backups. Then I did one last Time Machine backup and shut the Macbook down (running Leopard).

Now I swapped in the 500GB drive. Put in the snow leopard DVD, held down ‘c’ and let it boot up off the DVD. Then formatted the drive, and installed Snow Leopard (I picked a different account name than the one I used on my Leopard install, as Migration Assistant will complain later on if you try to restore into the account you are currently logged in as). Once I had SL installed, I then did a ‘connect to server’ in finder to mount my Samba share, then you need to click on the sparsebundle that you see, and your time machine backup should now be mounted. Now you can run ‘Migration Assistant’. Tell it you want to restore from a ‘Time Machine’ backup, then you should see your time machine backup to choose, then you get some screen with a few tick boxes and you have to wait while it says ‘Calculating’. I just left everything ticked and let it chug away restoring (it takes some time). Eventually, I just restarted, logged in as the username I used to use and voila it all looked suspiciously like my old Leopard setup.

The first thing I wanted to make sure was working was Mail. I started it up and it gives you some messages saying it needs to re-import everything and away it goes. Now, I use Mail.app as my main mail client, and I have an IMAP server that it connects to. I have some anal security in place for the IMAP server, so when Mail was restarted after the re-import I didn’t let it connect properly to the IMAP server. I then started going through my Inbox to see whether it looked OK. Everything seemed to be there, but I had a weird problem when I clicked on attachments. They would never load. There’d be some delay and then a popup highlighting there was a permission problem. This led to much trawling through Apple discussion threads and trying various things, but I could not get attachments to display. I just received that error instead

So I switched back to the original 160GB drive for a while. Then I tried again, clean format of the 500GB drive, installed SL, then this time I upgraded to 10.6.2, then did the Migration Assistant restore. No luck. Still had the same permission problem.

So by this time I was getting frustrated with Snow Leopard, so I thought I’d try to just put Leopard on the 500GB drive. Rather than use something sane like Carbon Copy Cloner or similar I thought I’d see how good a full restore from ‘Time Machine’ would go onto the 500GB drive. So with the old 160GB drive containing leopard, I did another Time Machine backup to the Samba share, shut down, swapped in the 500GB drive and booted off the Leopard install DVD that came with the macbook. Booted off the DVD, formatted the 500GB drive, then chose the option to restore from backup. This is where I learnt that the Apple install DVDs cannot mount a Samba share (or any kind of smb based share). They can mount a afp share though. So I thought, I’ll install netatalk on my Debian Lenny server that was running Samba, and just get it to share out the same mount that Samba was sharing.

This is when I discovered that the Debian Lenny version of netatalk cannot deal with encrypted passwords. Guess what OSX will only connect to? Yep, it only connects to a netatalk server that can do encrypted passwords (there is actually a defaults ‘switch’ that will allow you to connect to a netatalk user ; defaults write com.apple.AppleShareClient afp_cleartext_allow -bool true, but alas I could not get that to work from the shell environment offered on the install DVD), so then I ended up recompiling netatalk on Debian Lenny to handle encrypted passwords, which goes somehting like this;

$ sudo apt-get install libdb-dev
$ sudo apt-get install cdbs
$ sudo apt-get install libcrack2-dev
$ sudo apt-get install dh-buildinfo

$ cd debtmp
$ mkdir debtmp
$ sudo apt-get source netatalk
$ sudo apt-get install libcups2-dev
$ sudo apt-get build-dep netatalk
$ cd netatalk-2.0.3
$ sudo DEB_BUILD_OPTIONS=openssl fakeroot debian/rules binary
$ sudo dpkg -i ../netatalk_2.0.3-11+lenny1_amd64.deb

sudo afppasswd -ac
sudo afppasswd myuser

You might need to tweak some of the other files under /etc/netatalk too.

OK, so now using the notes on mounting here; I could mount my Time Machine backup from the Leopard install DVD;

mkdir /Volumes/tmachine
mount -t afp afp://username:password@server.hostname/tmachine /Volumes/tmachine

So then I kicked off my restore. Maybe 5 or 10 mins later the restore had encountered an error. I honestly can’t remember what it was. The discussion threads I found when googling for the error hinted that it has something to do with using the wrong install DVD (ie. one that came with a different Mac). I was definitely using the DVD that came with my Macbook, but I thought since I had the Snow Leopard DVD as well, then surely a full restore of a Leopard backup using the Snow Leopard boot DVD would end up with a Leopard system. So I did that, mounted the afp share OK, then started the restore … which continued just fine. Odd.

So that chugged away, and eventually I had a Leopard system on a bigger drive. I opened Mail.app, and again it said it needed to reimport Mail (just like for Snow Leopard). It chugged away for a bit, and then I had my Inbox open and it looked intact. I then found some of the messages with attachments I had had trouble with before and tried to open them. Instead of getting a permission error, I received no message at all … and in fact nothing happened (ie. my attachment still did not display). Hmmmm. Odd. It’s about now that I somehow realised that by blocking access to my IMAP server, I was preventing it from grabbing the attachment. This is probably obvious, but I imagine that Mail.app downloads attachment when you first click on them, and then caches them in the ~/Library/Mail\ Downloads directory, so that normally you’d be viewing the cached copy. However, I know that Time Machine ignores most cache directories in order to save space … so after my ‘fresh’ install plus restore, the cache directory was empty. I guess I was able to confirm my theory quickly by simply allowing my connection to my IMAP server to work (I noticed after I did this that there is a lot of activity in the ‘Activity’ window regarding caching etc.)

So after all that hoopla about Mail attachments, I thought I’ll reinstall Snow Leopard again. So same deal as before; boot off the SL DVD, format the 500GB drive, let it install SL, get it booted, perhaps upgrade to 10.6.3 or whatever, then use the Migration Assistant to restore from my Time Machine backup. This time when I ran Mail.app I let it connect to my IMAP server, and attachments worked as expected. All good

Apart from my funny Mail.app issue (which was sort of my fault), I was quite impressed with the whole Time Machine backup/restore. I am often asked by family or friends about ‘backing up their Windows PC’, and well there are a bunch of ways to back things up … but I always think a backup is only as good as your ability to restore it. My thoughts were that I could probably talk someone through doing a full restore of a Mac using a Time Machine backup … over the phone (maybe not from my convoluted AFP/Samba shared Time Machine server, but from a local USB drive). I am sure there are Windows products that claim to do something similar, but an enticing thing is that Time Machine is pretty much supported ‘out of the box’.

My upgrade was not all good. I have several vmware fusion guests on my Macbook, and when these were restored on the 500GB drive, all of them seemed to have very scary filesystem errors (the ones you get when there’s been some serious corruption). I thought maybe this was to do with Time Machine, but I ended up using my crazy rsync backups to restore from as well … and I still had the same filesystem errors. In the end, I booted back up off the 160GB drive in the macbook, and went through every Vmware guest and told them to power down (pretty much all of them were just ‘suspended’), then I did my rsync backup again, and restored them with the 500GB drive inside. And now they all work and have no filesystem errors. Obviously there is some strange inconsistency that occurs with the guest ‘state’ information during the upgrade to Snow Leopard.

Well it turns out you can ‘revert to snapshot’, but you just have to use a non-standard tool and some dmsetup commands. The tool is called dm-merge. Go to the dm-merge git repository or download the latest dm-merge snapshot . dm-merge just looks at the ‘changed blocks’ and pushes the old copies of them back into the original logical volume. You’ll need to compile the dm-merge program (just do a ‘make’). dm-merge comes with some doco. It explains two possible methods; reverting with an active logical volume OR reverting with an inactive one. In the former case you’re attempting to revert to a snapshot when the filesystem you’re trying to change is already mounted. I imagine that to be a very dangerous activity (but I guess it works well enough if there is little filesystem activity). I chose the latter (safer) method.

Here are my notes on the procedure. If this were a document for work, I would a) tell people they were crazy to even attempt this and b) if you’re still crazy make sure you make a backup. At least try this procedure out on a test machine first. The possibility of screwing your disk by virtue of a small typing mistake is quite high.

In the example below, I had a logical volume /dev/vgz/lvubuntu910full32bit that I was using as my hda drive in a KVM guest machine, and I had created a snapshot ‘snap1′ beforehand;

   lvcreate --size 1G --snapshot --name snap1 /dev/vgz/lvubuntu910full32bit

One important thing I realised recently is that the amount of space you allocate to a snapshot (eg. 1GB in the example above) can actually be resized without killing the snapshot (eg. lvresize -L 20G /dev/vgz/snap1)

Most of the following is a rehash of what is in the README that comes with dm-merge. I’m only referring to the method involving an inactive logical volume

# Firstly, shut down access to your logical volume (ie. /dev/vgz/vgzlvubuntu910full32bit).
   Since this is a KVM guest, all I did was shut down the guest. If you're doing this on a
   mounted logical volume then you should umount it.

# Duplicate the tables of the logical volumes and snapshot storage (note the use of the
   special -real and -cow suffixes)

dmsetup table vgz-lvubuntu910full32bit-real |dmsetup create tmplv
dmsetup table vgz-snap1-cow |dmsetup create tmpcow

# Deactivate the main lv

lvchange -a n /dev/vgz/lvubuntu910full32bit

# flush buffers

blockdev --flushbufs /dev/mapper/{tmplv,tmpcow}

# Do a dm-merge test run. You see lots of lines that start with 'dd' and a summary at the bottom.

dm-merge -i /dev/mapper/tmpcow -o /dev/mapper/tmplv -vd
...
dd of="/dev/mapper/tmplv" seek=2135444 if='/dev/mapper/tmpcow' iflag=direct skip=8275 count=1 bs=8b
dd of="/dev/mapper/tmplv" seek=2135445 if='/dev/mapper/tmpcow' iflag=direct skip=8276 count=1 bs=8b
dd of="/dev/mapper/tmplv" seek=2135446 if='/dev/mapper/tmpcow' iflag=direct skip=8277 count=1 bs=8b
dd of="/dev/mapper/tmplv" seek=5056466 if='/dev/mapper/tmpcow' iflag=direct skip=8278 count=1 bs=8b
dd of="/dev/mapper/tmplv" seek=5056467 if='/dev/mapper/tmpcow' iflag=direct skip=8279 count=1 bs=8b
Found 8246 exceptions of chunksize 4096, total size 33775616 bytes (32984 KiB, 32.211 MiB, 0.031 GiB).

# Do it for real. You get a couple of lines of output plus the same summary line.

 dm-merge -i /dev/mapper/tmpcow -o /dev/mapper/tmplv -f

Artificial sleep (1 second)
Found a proper MAGIC header: 0x70416e53
valid = 1
version = 1
chunk_size = 8 (4096 bytes)
Found 8246 exceptions of chunksize 4096, total size 33775616 bytes (32984 KiB, 32.211 MiB, 0.031 GiB).

# Flush buffers again

blockdev --flushbufs /dev/mapper/{tmplv,tmpcow}

# Remove the tmp lv's. These are just references to data, so it won't delete your real data.

dmsetup remove tmplv
dmsetup remove tmpcow

# Activate it

lvchange -a y /dev/vgz/lvubuntu910full32bit

Now try to start your KVM guest again … or remount it to check it out and confirm that it’s reverted to the original state. If you snapshotted a live KVM guest or a mounted filesystem, then you may find that your KVM guest will do a filesystem check on boot (fair enough), or you may need to run fsck on your filesystem before you can mount it.

Don’t forget to remove the snapshot you initially made (eg. lvremove /dev/vgz/snap1)

One thing to note is the ‘revert to snapshot’ capability has been ‘coming’ for a while in mainstream LVM. There’s a recent mention on LWN about it

Anyway, I have Debian Lenny on the T42 with several OpenVZ containers for things like Asterisk and a mercurial server. In addition it’s a streaming server, as I have two internal drives plus a large external 3.5″ USB drive.

The T42 is getting old, and I’d like to be able to run KVM on it … because basically I like KVM, but KVM needs a VT capable CPU. So I kept my eye out for a cheap 2nd hand VT capable laptop. I ended up buying a 2nd hand Thinkpad R60. This is probably one of the ugliest Thinkpads ever made which might explain the good price I got it for. I certainly wouldn’t buy one to lug around, but sitting on a desk at home quietly running a number of VMs is all I really wanted. It has a T2400 core duo (not core 2) cpu. Technically it is ‘VT capable’, but its a bit of an odd one (which probably applies to the T2500, T2600 etc) in that its not a 64 bit CPU like most other VT cpus. Other than being ugly and having an odd CPU, this R60 does have a lot going for it; 3GB of RAM, DVD burner, firewire, wifi, bluetooth, cardbus and expresscard slots (so I could add an esata expresscard device to add some highspeed desktop drives later on). And the HDD caddies I had for my T42 also fit in the R60.

I originally tried Ubuntu 9.10 on the R60, but the version of qemu they use in combination with KVM actually does not work properly on this old T2400 CPU. I could seemingly install a guest OS, but after reboot I had errors galore. Something to do with the ‘No execute’ bit I think. Anyway, this was yet another case of ubuntu quickly irritating me, so I ended up installing something else. Back to Debian Lenny. And of course VM’s work fine under Lenny and it’s older KVM and older qemu tools.

So far I’ve just run debian lenny 32 bit and Windows 2008 R2 32 bit in some KVM VMs. Both seem to run quite well. The laptop has 3GB of ram, so I can run quite a few VMs at the same time OK. I’ve migrated my asterisk installation from the openVZ container it was in on my T42 to a KVM VM on the R60. It works quite well.

One thing that does bug me about using KVM is the lack of easy to use snapshots. ESXi spoils you in the simplicity of it’s snapshots on running or shutdown VMs. You can use them for backups or easily roll back to them. However, KVM has a haphazard collection of tools that don’t quite do the same thing. virt-manager offers no option for snapshots. If you use virsh instead it does have a ‘save’ option to snapshot a machine … but it seems to always shut the machine down when it runs. If you don’t use libvirtd to manage your VMs and instead use kvm directly, then you can interact directly with the qemu monitor of a VM and it has a savevm option which does something akin to a snapshot, but everytime I’ve tried it on a live VM, it pauses the VM for several minutes which is somewhat annoying. The closest thing to ESXi snapshots that I can think of is to use logical volumes for the VM’s virtual disk, then use LVM snapshots to create snapshots of them. These aren’t full ‘state’ snapshots that include a ram snapshot etc, but are probably good enough for my purposes. The tricky part is rolling back to a snapshot which is not really supported with logical volumes out of the box, but you can do it with something like the dm-merge tool.

I also tried Fedora 12 on the R60. The motivation was that the version of KVM on Debian Lenny is old, and Fedora 12 has quite a number of new features of KVM including better qcow2 performance and something that looks for identical memory pages in VMs to reduce memory usage. This latter feature appears as some ksmd daemon that seems to hog 50% of one CPU. Very annoying, so I killed it. I was hoping that the qcow2 enhancements might solve my ‘snapshots’ issue, but so far it doesn’t look like it. I need to investigate more. In the meantime, I’ll go back to Debian Lenny on the R60.

Of course, I did try running ESXi on the R60. ESXi 3.5u4 seemingly worked perfectly on it. I didn’t't need any fancy DIY network drivers. It just picked up the internal broadcom gigabit and was able to see the two internal drives. The only problem is that I get a Purple screen of death when I try do a shutdown (some error to do with ACPI, and setting acpi=off in the kernel options does not help). However, if you tell ESXi to reboot it reboots as expected (UPDATE: I have just tried ESXi 3.5u5 … and is has the same purple screen when you attempt a shutdown). I didn’t bother trying ESXi 4, as I am pretty sure that needs a 64 bit CPU.

The motherboard was a Asus P5LD2 which was one of the few ‘cheaper’ socket 775 boards available at the time that were compatible with the (then) new Core 2 Duos. It’s an interesting board as it has three IDE channels (ie. up to 6 IDE drives) and four SATA II interfaces. It uses a 945P chipset which doesn’t have a great reputation for overclocking. However, I could get the E6300 to run at 2.33GHz reliably which was ok.

More recently I’ve added more ram to the board ( four 1GB DIMMS in total) and discovered the problem of some of these older socket 775 chipsets. They can really only have 3GB maximum RAM. I thought this was just a 32bit OS problem, but 64 bit OS’s have the same problem with this board. Basically the chipset can only address 4GB of memory, and quite large chunks of memory are mapped for various IO purposes, which is made worse by certain types of cards (I think I get 3.5GB if I pull out my TV tuner card, 3 or 3.2GB with it plugged in). I know some older motherboards have an option for remapping a memory hole … but even on the latest BIOS, I don’t have such an option.

Also, the board is limited to mostly older core 2 duo chips, and quad cores are definitely a no go.

So, like any tech nutter, I’ve been researching the best ‘bang for buck’ upgrade. All the Intel socket 1156 i5/i7 chips came out a month or two ago and they do indeed look quite interesting. The i5 750 is quad core, but all the i7 models have a newer form of hyperthreading so it is a bit like getting ’8 cores’. To upgrade my system to i5/i7 there is quite a bit of expense (close to NZ$1000 just for an i7 860, motherboard and 4GB of ram), so I wondered exactly how much faster they really are. A good tool I found is the anandtech bench (beta).

With the Anandtech bench you can compare benchmarks between any two CPUs that they happen to have reviewed. A good test I thought would be to compare the i5 750 against an older quad core like the Q8400. The i5 is definitely faster in most tests, but in some of the tests that I’m interested in (like the x264 tests) it’s not ‘amazingly’ faster.

So feeling quite ‘cheap’ I thought I’d just upgrade my motherboard to a newer socket 775 board. That way I could keep my DDR2 ram, keep my core2duo and immediately get a full 4GB of ram, plus the ability to get a quad core core2duo later on. Sounded reasonable at the time. So I did some research and ended up coming home with a Gigabyte EP45-UD3LR board.

Supposedly the EP45-UD3LR can handle 16GB of ram … but until someone makes cheap 4GB DDR2 DIMMs, I’ll be happy with 4GB. It’s a full ATX board and has no onboard gfx. Just sound and the usual plethora of usb and sata ports. Now, I usually run linux on this system, but a secondary reason for getting this board is that these ‘EP45′ boards are very popular in the Hackintosh community for turning regular PC’s into ‘hacked’ OSX systems. I don’t really need another Mac, but all this hackintosh stuff is quite an interesting diversion in my pursuit of time wasting activities.

So for the past few weekends I’ve been playing with installing Snow Leopard on this thing (I actually bought the retail DVD for it). It’s still a far from simple process, that requires reading copious Howto’s and forum posts on InsanelyMac and other related sites. There are a number of complications; a) All the Gigabyte EP45 boards are slightly different, so while lifehacker.com might publish some nice howto’s for someone using a EP45-UD3P motherboard. My EP45-UD3LR is not ‘quite’ the same b) There are different ‘approaches’ you can take that have bizarre names that unless you’ve been trawling forums for quite some time you won’t understand; you can use hacked kexts, you can modify the DSDT, you can ‘inject’ EFI strings and seemingly you can use a combination of these .. though I’m not entirely sure. I think what’s happened is that the hackintosh community is obviously trying to increase the probability that when you do an Apple ‘Software Update’ that includes a kernel update (eg 10.6 to 10.6.1) that your machine doesn’t die painfully. I got the idea that modding the DSDT instead of injecting EFI strings somehow makes things ‘better’ and a big goal is to reduce the necessity for hacked kexts.

So does it work? Yes it does. A lot better than I thought it would even though I have had intermittent crashes (the screen goes dark and you get a message telling you to hold in the power button) which I think I’ve finally resolved (my Nvidia card seems to hate the 64 bit kernel drivers, but works OK booting in 32 bit mode). I won’t go into all the detail of what I did to build the thing, but here are my key points;

• I used the lifehacker.com ‘How to build a Hackintosh with Snow Leopard Start to Finish’ article (the original article, not the followup)
• I also referred to this EP45-UD3R guide on insanelymac. Especially the bits about setting the UUID in /Extra/smbios.plist and the PlatformUUID.kext. I found I couldn’t install Chameleon onto the hard drive until I got the UUID fixed
• I have an old GeForce 7300LE card with 128MB of RAM. This card is not listed in EFIStudio, but originally I just selected GeForce 7300 GT 256MB which seemed to work OK. I also noticed the Front Row would not work at all unless I had something like the following in /Extra/com.apple.Boot.plist:

      <key>Graphics Mode</key>
      <string>1280x1024x32</string>

• I used the ‘SL Pack’ that is referred to on various insanelymac howtos, especially the Extras folder it tells you to start with. ie. I didnt use the one that the LifeHacker article refers to. Though I did originally use the lifehacker one. I have no idea if there is any difference.
• To get my ALC888 sound to work required quite a few steps. There’s a really useful comment by Dith close to the bottom of this forum page. I followed the steps that start with ‘Redid my whole Snow Leopard installation’. And then I grabbed the LegacyHDA.kext that is at this path in the SL Pack and copied it into /Extra/Extensions replacing the one I already had.;

  SL Pack/DSDT Stuff/How to Patch DSDT /series of LegacyHDA 888 (ALC888)/3out2in HDA headphone/LegacyHDA.kext

• I manually enable QE/CI (well I think I did) by sudo’ing to root and running;

  defaults write /Library/Preferences/com.apple.windowserver GLCompositor -dict tileHeight -int 256 tileWidth -int 256

• I tended to run ‘Kext Utility’ between a lot of the above steps. Never sure if it makes much difference.
• I later found that OSX86Tools can generate the EFI string for my GeForce 7300LE 128MB card. So I used that in my /Extra/com.apple.Boot.plist hoping that it would fix my intermittent crash problem. It didn’t fix the crashes, but it did make me feel warm and fuzzy.
• I found that I could consistently crash the machine by running ‘Chess’ and just moving the mouse. Obviously Apple knows that I’m not very good at Chess, and was saving me some misery … These crashes were with a 64 bit kernel. Then I tried booting with a 32 bit kernel and Chess works fine, and so far no crashes (though I still can’t play Chess very well). I’m using Chameleon RC3 and had to put ‘arch=i386′ in the /Extra/com.apple.Boot.plist file like so;

          <key>Kernel Flags</key>
  	<string>arch=i386</string>

I’ve since updated from 10.6 to 10.6.1 with no dramas. The upgrade to 10.6.2 had some minor hitches in that you need to remove the older SleepEnabler.kext. See here for more info.

And of course, the new motherboard works fine running linux too!

It took me a while to get 2.6.31 to boot properly. I have an older nvidia card in this system, and the default setup had the kernel loading the nvidiafb module … which does not seem to work with my system (I just get a black or gray screen). Fortunately I could still ssh in from another box. It took me a while to work out how to properly disable nvidiafb, which ended up being somehting like;

cd /etc
mv modules.conf modules.conf.orig
cd modprobe.d
# Edit the blacklist file and add 'blacklist nvidiafb' to the end.
vi blacklist
# now update the initrd
update-initramfs -u

Also, in order to get X windows working I ended up downloading the driver from nvidia.com ( I used NVIDIA-Linux-x86_64-185.18.36-pkg2.run).

Once I had it booted (properly) off 2.6.31, I thought I’d have a look at btrfs. It’s a new filesystem for linux which is meant to be a lot like ZFS, but it’s not quite there yet in the stability department. However, I thought I read somewhere that the version included in 2.6.31 was ‘pretty good’, so it was worth a look. What I wanted to try was to have a btrfs root partition.

The first thing was to recompile my kernel again. Btrfs seemed to be turned off by default if you’re doing the make-kpkg stuff. So my setup process was something like;

cd /usr/src
tar xvjf /tmp/linux-2.6.31.tar.bz2
mv linux linux.orig
ln -s linux-2.6.31 linux
cd linux
make-kpkg clean
make menuconfig  <--- make sure you go to filesystems and enable btrfs as a module)
fakeroot make-kpkg --initrd --append-to-version=-custom2 kernel_image kernel_headers
# Wait 10 years
cd ..
dpkg -i linux-headers-2.6.31-custom2_2.6.31-custom2-10.00.Custom_amd64.deb
dpkg -i linux-image-2.6.31-custom2_2.6.31-custom2-10.00.Custom_amd64.deb

Now my plan was to make a copy of my root fs into a btrfs partition and boot off that. I have /boot as an ext2 partition, and my normal lenny root fs is on a logical volume as an ext3 fs. I thought I'd take a snapshot of the ext3 rootfs and copy it into a btrfs logical volume and boot off that. But the first thing to do is to get the btrfs user space tools;

git clone git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs-unstable.git
cd btrfs-progs-unstable
make
make install

Then setup the logical volumes and copy the data (NB: You might have to do a 'modprobe libcrc32c ; modprobe btrfs' first). For reference, I use a volume group called vgz which will undoubtedly be different to whatever system you might have.

mkdir /tmp/src
mkdir /tmp/dest
lvcreate -L 1G -s -n snap1 /dev/vgz/lvlenny64root
mount /dev/vgz/snap1 /tmp/src
lvcreate -L 12G -n lvlenny64btrfsroot /dev/vgz
mkfs.btrfs /dev/vgz/lvlenny64btrfsroot
mount -t btrfs /dev/vgz/lvlenny64btrfsroot /tmp/dest
cd /tmp/src
find . -print | cpio -dumpv /tmp/dest
cd /
# You'll need to edit the fstab to change the root device and filesystem type. For example:
#    /dev/mapper/vgz-lvlenny64btrfsroot /               btrfs    errors=remount-ro 0       1
vi /tmp/dest/etc/fstab
umount /tmp/dest
lvremove /dev/vgz/snap1
umount /tmp/src

Now we need to convince your initrd to load the btrfs modules (this is debian/ubuntu/whatever specific). I edited /etc/initramfs-tools/modules and added these lines in;

libcrc32c
zlib_deflate
crc32c
btrfs

And then you need to run update-initramfs -u
At this point I had to do some fiddling with /boot/grub/menu.lst. I just made an extra stanza in the file that looked like;

title           BTRFS Debian GNU/Linux, kernel 2.6.31-custom2
root            (hd1,1)
kernel          /vmlinuz-2.6.31-custom2 root=/dev/mapper/vgz-lvlenny64btrfsroot rootfstype=btrfs ro
initrd          /initrd.img-2.6.31-custom2

Obviously don't just copy this for your own setup, as the root line will be different and possibly some other stuff. Admitedly, I am probably missing something with this setup as when I go to boot off my btrfs root partition, I get kernel boot message up to the point where it's mounting the root filesystem, then just sits there for 5 minutes before eventually mounting it ok and continuing on. It's all a bit odd (UPDATE: This long pause is to do with the initramfs fstype and/or vol_id tools not being able to recognise btrfs filesystems. Eventually someone will update those tools, but until then you might want to edit /usr/share/initramfs-tools/scripts/local, hack the get_fstype function and then run update-initramfs-u. I'll put some notes at the bottom of the post).

Anyway, so I got a bootable system with btrfs. The first thing I wanted to try was snapshots, so I duely did something like;

btrfsctl -s snap1 /

When I first did that I thought it would create a snap1 directory under the / (ie. root) of my root fs. It didn't. It actually created a snap1 directory in the directory I was currently in. So within that directory was an intact snapshot of my root filesystem. Of course the first thing you want to do is delete the snapshot ... which in 2.6.31 is a 'future feature'. You can try and rm -rf the snapshot directory ... but I always got some circular file reference errors and it took ages as well. In fact I took a few snapshots as I played around, then left it running over night. In the morning it told me my filesystem was full and the hard drive light was permanantly on. Odd.

Fortunately, the ability to delete snapshots has actually been added to btrfs. You just need an even more recent kernel ... or what I did was to just check out the btrfs kernel tree (I went to here and clicked 'tree' up the top, then 'fs', then 'btrfs' then the 'snapshot' button near the top) and shove it into my 2.6.31 kernel and go through the whole make-kpkg thing again and try booting again (actually I did a few more steps and just deleted my whole btrfs example logical volume and created a new one with no snapshots).

Now with my '2.6.31 and a bit' kernel I can make a snapshot (I have now learned that putting a leading / in front of the snapshot name is probably sensible);

btrfsctl -s /snap1 /

And so if I cd /snap1 I can access my snapshot. But more importantly I can delete it;

btrfsctl -D snap1 /

The arguments to the 'btrfsctl -D' seem to be the snapshot name followed by the directory in which it was made.

I also found this post on root'ing the net that shows how you can mount the snapshot using the mount command (ie. in addition to the snapshot appearing as a subdirectory);

mount -o subvol=snap1 /dev/vgz/lvlenny64btrfsroot /mnt

Most stuff on btrfs indicates that subvols and snapshots are treated the same. If you create a new snapshot you get a copy of what you snapshotted. Creating a new subvol gives you an empty directory which I assume can be controlled separately, but uses the same pool of disk resources of my main btrfs filesystem. I'm not entirely sure. There is an option to the btrfsctl command which can resize a 'something' but I'm not too sure exactly what it does. I tried resizing down and df doesn't reflect any difference, but doing a btrfs-show did show some change. At the moment there aren't really any commands to list snapshots or show usage. There is a btrfs-debug-tree command that spits out reams of info ... and does mention my snapshots deep within it.

btrfs can apparently do RAID1, RAID0 and RAID10. I like that redundancy is a feature of the filesystem. I've never really understood why LVM2 for linux has no redundancy features (and the mirroring capability it does have is not very useful). I haven't tried any of the btrfs RAID features yet.

So that's all I've looked at so far. One of the key features of ZFS that I'd really like to see in btrfs is 'rollback to snapshot'. This is such an incredibly useful feature in a fast paced IT environment. Given that btrfs is a copy-on-write filesystem, I am hoping they put rollback in at some point.

Honestly I haven't tested it enough to determine how stable it is. It seems fine so far, but this is not a server type machine.

UPDATE: re getting rid of the long pause on boot.
I edited /usr/share/initramfs-tools/scripts/local, and added the extra lines shown below. Be warned, this is an awful 'hack' that assumes that any filesystem type that is not recognised is 'btrfs'. Once you've updated the file you'll need to run update-initramfs -u

get_fstype ()
{
        local FS FSTYPE FSSIZE RET
        FS="${1}"

        # vol_id has a more complete list of file systems,
        # but fstype is more robust
        eval $(fstype "${FS}" 2> /dev/null)
        if [ "$FSTYPE" = "unknown" ] && [ -x /lib/udev/vol_id ]; then
                FSTYPE=$(/lib/udev/vol_id -t "${FS}" 2> /dev/null)
        fi
        RET=$?

        if [ -z "${FSTYPE}" ]; then
                FSTYPE="unknown"
        fi

# Hack for my btrfs root. Terrible hack
        if [ "${FSTYPE}" = unknown ];then
                FSTYPE=btrfs
                RET=0
        fi
# End of my hack

        echo "${FSTYPE}"
        return ${RET}
}