Given that I had a new (larger) drive. I thought I’d play with how well Time Machine and the Migration Assistant work. So as a test I setup a large mount point on a linux box and shared it out using Samba to the Macbook, then did the defaults command below to enable a samba share as a Time Machine destination:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

I also did the mucking around with hdiutil to create the necessary sparsebundle (if you google for ‘TMShowUnsupportedNetworkVolumes hdiutil’ you’ll find a heap of references). It takes a long time to do the first backup. I then set it to ‘manual’ mode so it’s not doing continual backups. Then I did one last Time Machine backup and shut the Macbook down (running Leopard).

Now I swapped in the 500GB drive. Put in the snow leopard DVD, held down ‘c’ and let it boot up off the DVD. Then formatted the drive, and installed Snow Leopard (I picked a different account name than the one I used on my Leopard install, as Migration Assistant will complain later on if you try to restore into the account you are currently logged in as). Once I had SL installed, I then did a ‘connect to server’ in finder to mount my Samba share, then you need to click on the sparsebundle that you see, and your time machine backup should now be mounted. Now you can run ‘Migration Assistant’. Tell it you want to restore from a ‘Time Machine’ backup, then you should see your time machine backup to choose, then you get some screen with a few tick boxes and you have to wait while it says ‘Calculating’. I just left everything ticked and let it chug away restoring (it takes some time). Eventually, I just restarted, logged in as the username I used to use and voila it all looked suspiciously like my old Leopard setup.

The first thing I wanted to make sure was working was Mail. I started it up and it gives you some messages saying it needs to re-import everything and away it goes. Now, I use Mail.app as my main mail client, and I have an IMAP server that it connects to. I have some anal security in place for the IMAP server, so when Mail was restarted after the re-import I didn’t let it connect properly to the IMAP server. I then started going through my Inbox to see whether it looked OK. Everything seemed to be there, but I had a weird problem when I clicked on attachments. They would never load. There’d be some delay and then a popup highlighting there was a permission problem. This led to much trawling through Apple discussion threads and trying various things, but I could not get attachments to display. I just received that error instead

So I switched back to the original 160GB drive for a while. Then I tried again, clean format of the 500GB drive, installed SL, then this time I upgraded to 10.6.2, then did the Migration Assistant restore. No luck. Still had the same permission problem.

So by this time I was getting frustrated with Snow Leopard, so I thought I’d try to just put Leopard on the 500GB drive. Rather than use something sane like Carbon Copy Cloner or similar I thought I’d see how good a full restore from ‘Time Machine’ would go onto the 500GB drive. So with the old 160GB drive containing leopard, I did another Time Machine backup to the Samba share, shut down, swapped in the 500GB drive and booted off the Leopard install DVD that came with the macbook. Booted off the DVD, formatted the 500GB drive, then chose the option to restore from backup. This is where I learnt that the Apple install DVDs cannot mount a Samba share (or any kind of smb based share). They can mount a afp share though. So I thought, I’ll install netatalk on my Debian Lenny server that was running Samba, and just get it to share out the same mount that Samba was sharing.

This is when I discovered that the Debian Lenny version of netatalk cannot deal with encrypted passwords. Guess what OSX will only connect to? Yep, it only connects to a netatalk server that can do encrypted passwords (there is actually a defaults ’switch’ that will allow you to connect to a netatalk user ; defaults write com.apple.AppleShareClient afp_cleartext_allow -bool true, but alas I could not get that to work from the shell environment offered on the install DVD), so then I ended up recompiling netatalk on Debian Lenny to handle encrypted passwords, which goes somehting like this;

$ sudo apt-get install libdb-dev
$ sudo apt-get install cdbs
$ sudo apt-get install libcrack2-dev
$ sudo apt-get install dh-buildinfo

$ cd debtmp
$ mkdir debtmp
$ sudo apt-get source netatalk
$ sudo apt-get install libcups2-dev
$ sudo apt-get build-dep netatalk
$ cd netatalk-2.0.3
$ sudo DEB_BUILD_OPTIONS=openssl fakeroot debian/rules binary
$ sudo dpkg -i ../netatalk_2.0.3-11+lenny1_amd64.deb

sudo afppasswd -ac
sudo afppasswd myuser

You might need to tweak some of the other files under /etc/netatalk too.

OK, so now using the notes on mounting here; I could mount my Time Machine backup from the Leopard install DVD;

mkdir /Volumes/tmachine
mount -t afp afp://username:password@server.hostname/tmachine /Volumes/tmachine

So then I kicked off my restore. Maybe 5 or 10 mins later the restore had encountered an error. I honestly can’t remember what it was. The discussion threads I found when googling for the error hinted that it has something to do with using the wrong install DVD (ie. one that came with a different Mac). I was definitely using the DVD that came with my Macbook, but I thought since I had the Snow Leopard DVD as well, then surely a full restore of a Leopard backup using the Snow Leopard boot DVD would end up with a Leopard system. So I did that, mounted the afp share OK, then started the restore … which continued just fine. Odd.

So that chugged away, and eventually I had a Leopard system on a bigger drive. I opened Mail.app, and again it said it needed to reimport Mail (just like for Snow Leopard). It chugged away for a bit, and then I had my Inbox open and it looked intact. I then found some of the messages with attachments I had had trouble with before and tried to open them. Instead of getting a permission error, I received no message at all … and in fact nothing happened (ie. my attachment still did not display). Hmmmm. Odd. It’s about now that I somehow realised that by blocking access to my IMAP server, I was preventing it from grabbing the attachment. This is probably obvious, but I imagine that Mail.app downloads attachment when you first click on them, and then caches them in the ~/Library/Mail\ Downloads directory, so that normally you’d be viewing the cached copy. However, I know that Time Machine ignores most cache directories in order to save space … so after my ‘fresh’ install plus restore, the cache directory was empty. I guess I was able to confirm my theory quickly by simply allowing my connection to my IMAP server to work (I noticed after I did this that there is a lot of activity in the ‘Activity’ window regarding caching etc.)

So after all that hoopla about Mail attachments, I thought I’ll reinstall Snow Leopard again. So same deal as before; boot off the SL DVD, format the 500GB drive, let it install SL, get it booted, perhaps upgrade to 10.6.3 or whatever, then use the Migration Assistant to restore from my Time Machine backup. This time when I ran Mail.app I let it connect to my IMAP server, and attachments worked as expected. All good

Apart from my funny Mail.app issue (which was sort of my fault), I was quite impressed with the whole Time Machine backup/restore. I am often asked by family or friends about ‘backing up their Windows PC’, and well there are a bunch of ways to back things up … but I always think a backup is only as good as your ability to restore it. My thoughts were that I could probably talk someone through doing a full restore of a Mac using a Time Machine backup … over the phone (maybe not from my convoluted AFP/Samba shared Time Machine server, but from a local USB drive). I am sure there are Windows products that claim to do something similar, but an enticing thing is that Time Machine is pretty much supported ‘out of the box’.

My upgrade was not all good. I have several vmware fusion guests on my Macbook, and when these were restored on the 500GB drive, all of them seemed to have very scary filesystem errors (the ones you get when there’s been some serious corruption). I thought maybe this was to do with Time Machine, but I ended up using my crazy rsync backups to restore from as well … and I still had the same filesystem errors. In the end, I booted back up off the 160GB drive in the macbook, and went through every Vmware guest and told them to power down (pretty much all of them were just ’suspended’), then I did my rsync backup again, and restored them with the 500GB drive inside. And now they all work and have no filesystem errors. Obviously there is some strange inconsistency that occurs with the guest ’state’ information during the upgrade to Snow Leopard.

Well it turns out you can ‘revert to snapshot’, but you just have to use a non-standard tool and some dmsetup commands. The tool is called dm-merge. Go to the dm-merge git repository or download the latest dm-merge snapshot . dm-merge just looks at the ‘changed blocks’ and pushes the old copies of them back into the original logical volume. You’ll need to compile the dm-merge program (just do a ‘make’). dm-merge comes with some doco. It explains two possible methods; reverting with an active logical volume OR reverting with an inactive one. In the former case you’re attempting to revert to a snapshot when the filesystem you’re trying to change is already mounted. I imagine that to be a very dangerous activity (but I guess it works well enough if there is little filesystem activity). I chose the latter (safer) method.

Here are my notes on the procedure. If this were a document for work, I would a) tell people they were crazy to even attempt this and b) if you’re still crazy make sure you make a backup. At least try this procedure out on a test machine first. The possibility of screwing your disk by virtue of a small typing mistake is quite high.

In the example below, I had a logical volume /dev/vgz/lvubuntu910full32bit that I was using as my hda drive in a KVM guest machine, and I had created a snapshot ’snap1′ beforehand;

   lvcreate --size 1G --snapshot --name snap1 /dev/vgz/lvubuntu910full32bit

One important thing I realised recently is that the amount of space you allocate to a snapshot (eg. 1GB in the example above) can actually be resized without killing the snapshot (eg. lvresize -L 20G /dev/vgz/snap1)

Most of the following is a rehash of what is in the README that comes with dm-merge. I’m only referring to the method involving an inactive logical volume

# Firstly, shut down access to your logical volume (ie. /dev/vgz/vgzlvubuntu910full32bit).
   Since this is a KVM guest, all I did was shut down the guest. If you're doing this on a
   mounted logical volume then you should umount it.

# Duplicate the tables of the logical volumes and snapshot storage (note the use of the
   special -real and -cow suffixes)

dmsetup table vgz-lvubuntu910full32bit-real |dmsetup create tmplv
dmsetup table vgz-snap1-cow |dmsetup create tmpcow

# Deactivate the main lv

lvchange -a n /dev/vgz/lvubuntu910full32bit

# flush buffers

blockdev --flushbufs /dev/mapper/{tmplv,tmpcow}

# Do a dm-merge test run. You see lots of lines that start with 'dd' and a summary at the bottom.

dm-merge -i /dev/mapper/tmpcow -o /dev/mapper/tmplv -vd
...
dd of="/dev/mapper/tmplv" seek=2135444 if='/dev/mapper/tmpcow' iflag=direct skip=8275 count=1 bs=8b
dd of="/dev/mapper/tmplv" seek=2135445 if='/dev/mapper/tmpcow' iflag=direct skip=8276 count=1 bs=8b
dd of="/dev/mapper/tmplv" seek=2135446 if='/dev/mapper/tmpcow' iflag=direct skip=8277 count=1 bs=8b
dd of="/dev/mapper/tmplv" seek=5056466 if='/dev/mapper/tmpcow' iflag=direct skip=8278 count=1 bs=8b
dd of="/dev/mapper/tmplv" seek=5056467 if='/dev/mapper/tmpcow' iflag=direct skip=8279 count=1 bs=8b
Found 8246 exceptions of chunksize 4096, total size 33775616 bytes (32984 KiB, 32.211 MiB, 0.031 GiB).

# Do it for real. You get a couple of lines of output plus the same summary line.

 dm-merge -i /dev/mapper/tmpcow -o /dev/mapper/tmplv -f

Artificial sleep (1 second)
Found a proper MAGIC header: 0x70416e53
valid = 1
version = 1
chunk_size = 8 (4096 bytes)
Found 8246 exceptions of chunksize 4096, total size 33775616 bytes (32984 KiB, 32.211 MiB, 0.031 GiB).

# Flush buffers again

blockdev --flushbufs /dev/mapper/{tmplv,tmpcow}

# Remove the tmp lv's. These are just references to data, so it won't delete your real data.

dmsetup remove tmplv
dmsetup remove tmpcow

# Activate it

lvchange -a y /dev/vgz/lvubuntu910full32bit

Now try to start your KVM guest again … or remount it to check it out and confirm that it’s reverted to the original state. If you snapshotted a live KVM guest or a mounted filesystem, then you may find that your KVM guest will do a filesystem check on boot (fair enough), or you may need to run fsck on your filesystem before you can mount it.

Don’t forget to remove the snapshot you initially made (eg. lvremove /dev/vgz/snap1)

One thing to note is the ‘revert to snapshot’ capability has been ‘coming’ for a while in mainstream LVM. There’s a recent mention on LWN about it

Anyway, I have Debian Lenny on the T42 with several OpenVZ containers for things like Asterisk and a mercurial server. In addition it’s a streaming server, as I have two internal drives plus a large external 3.5″ USB drive.

The T42 is getting old, and I’d like to be able to run KVM on it … because basically I like KVM, but KVM needs a VT capable CPU. So I kept my eye out for a cheap 2nd hand VT capable laptop. I ended up buying a 2nd hand Thinkpad R60. This is probably one of the ugliest Thinkpads ever made which might explain the good price I got it for. I certainly wouldn’t buy one to lug around, but sitting on a desk at home quietly running a number of VMs is all I really wanted. It has a T2400 core duo (not core 2) cpu. Technically it is ‘VT capable’, but its a bit of an odd one (which probably applies to the T2500, T2600 etc) in that its not a 64 bit CPU like most other VT cpus. Other than being ugly and having an odd CPU, this R60 does have a lot going for it; 3GB of RAM, DVD burner, firewire, wifi, bluetooth, cardbus and expresscard slots (so I could add an esata expresscard device to add some highspeed desktop drives later on). And the HDD caddies I had for my T42 also fit in the R60.

I originally tried Ubuntu 9.10 on the R60, but the version of qemu they use in combination with KVM actually does not work properly on this old T2400 CPU. I could seemingly install a guest OS, but after reboot I had errors galore. Something to do with the ‘No execute’ bit I think. Anyway, this was yet another case of ubuntu quickly irritating me, so I ended up installing something else. Back to Debian Lenny. And of course VM’s work fine under Lenny and it’s older KVM and older qemu tools.

So far I’ve just run debian lenny 32 bit and Windows 2008 R2 32 bit in some KVM VMs. Both seem to run quite well. The laptop has 3GB of ram, so I can run quite a few VMs at the same time OK. I’ve migrated my asterisk installation from the openVZ container it was in on my T42 to a KVM VM on the R60. It works quite well.

One thing that does bug me about using KVM is the lack of easy to use snapshots. ESXi spoils you in the simplicity of it’s snapshots on running or shutdown VMs. You can use them for backups or easily roll back to them. However, KVM has a haphazard collection of tools that don’t quite do the same thing. virt-manager offers no option for snapshots. If you use virsh instead it does have a ’save’ option to snapshot a machine … but it seems to always shut the machine down when it runs. If you don’t use libvirtd to manage your VMs and instead use kvm directly, then you can interact directly with the qemu monitor of a VM and it has a savevm option which does something akin to a snapshot, but everytime I’ve tried it on a live VM, it pauses the VM for several minutes which is somewhat annoying. The closest thing to ESXi snapshots that I can think of is to use logical volumes for the VM’s virtual disk, then use LVM snapshots to create snapshots of them. These aren’t full ’state’ snapshots that include a ram snapshot etc, but are probably good enough for my purposes. The tricky part is rolling back to a snapshot which is not really supported with logical volumes out of the box, but you can do it with something like the dm-merge tool.

I also tried Fedora 12 on the R60. The motivation was that the version of KVM on Debian Lenny is old, and Fedora 12 has quite a number of new features of KVM including better qcow2 performance and something that looks for identical memory pages in VMs to reduce memory usage. This latter feature appears as some ksmd daemon that seems to hog 50% of one CPU. Very annoying, so I killed it. I was hoping that the qcow2 enhancements might solve my ’snapshots’ issue, but so far it doesn’t look like it. I need to investigate more. In the meantime, I’ll go back to Debian Lenny on the R60.

Of course, I did try running ESXi on the R60. ESXi 3.5u4 seemingly worked perfectly on it. I didn’t't need any fancy DIY network drivers. It just picked up the internal broadcom gigabit and was able to see the two internal drives. The only problem is that I get a Purple screen of death when I try do a shutdown (some error to do with ACPI, and setting acpi=off in the kernel options does not help). However, if you tell ESXi to reboot it reboots as expected (UPDATE: I have just tried ESXi 3.5u5 … and is has the same purple screen when you attempt a shutdown). I didn’t bother trying ESXi 4, as I am pretty sure that needs a 64 bit CPU.

The motherboard was a Asus P5LD2 which was one of the few ‘cheaper’ socket 775 boards available at the time that were compatible with the (then) new Core 2 Duos. It’s an interesting board as it has three IDE channels (ie. up to 6 IDE drives) and four SATA II interfaces. It uses a 945P chipset which doesn’t have a great reputation for overclocking. However, I could get the E6300 to run at 2.33GHz reliably which was ok.

More recently I’ve added more ram to the board ( four 1GB DIMMS in total) and discovered the problem of some of these older socket 775 chipsets. They can really only have 3GB maximum RAM. I thought this was just a 32bit OS problem, but 64 bit OS’s have the same problem with this board. Basically the chipset can only address 4GB of memory, and quite large chunks of memory are mapped for various IO purposes, which is made worse by certain types of cards (I think I get 3.5GB if I pull out my TV tuner card, 3 or 3.2GB with it plugged in). I know some older motherboards have an option for remapping a memory hole … but even on the latest BIOS, I don’t have such an option.

Also, the board is limited to mostly older core 2 duo chips, and quad cores are definitely a no go.

So, like any tech nutter, I’ve been researching the best ‘bang for buck’ upgrade. All the Intel socket 1156 i5/i7 chips came out a month or two ago and they do indeed look quite interesting. The i5 750 is quad core, but all the i7 models have a newer form of hyperthreading so it is a bit like getting ‘8 cores’. To upgrade my system to i5/i7 there is quite a bit of expense (close to NZ$1000 just for an i7 860, motherboard and 4GB of ram), so I wondered exactly how much faster they really are. A good tool I found is the anandtech bench (beta).

With the Anandtech bench you can compare benchmarks between any two CPUs that they happen to have reviewed. A good test I thought would be to compare the i5 750 against an older quad core like the Q8400. The i5 is definitely faster in most tests, but in some of the tests that I’m interested in (like the x264 tests) it’s not ‘amazingly’ faster.

So feeling quite ‘cheap’ I thought I’d just upgrade my motherboard to a newer socket 775 board. That way I could keep my DDR2 ram, keep my core2duo and immediately get a full 4GB of ram, plus the ability to get a quad core core2duo later on. Sounded reasonable at the time. So I did some research and ended up coming home with a Gigabyte EP45-UD3LR board.

Supposedly the EP45-UD3LR can handle 16GB of ram … but until someone makes cheap 4GB DDR2 DIMMs, I’ll be happy with 4GB. It’s a full ATX board and has no onboard gfx. Just sound and the usual plethora of usb and sata ports. Now, I usually run linux on this system, but a secondary reason for getting this board is that these ‘EP45′ boards are very popular in the Hackintosh community for turning regular PC’s into ‘hacked’ OSX systems. I don’t really need another Mac, but all this hackintosh stuff is quite an interesting diversion in my pursuit of time wasting activities.

So for the past few weekends I’ve been playing with installing Snow Leopard on this thing (I actually bought the retail DVD for it). It’s still a far from simple process, that requires reading copious Howto’s and forum posts on InsanelyMac and other related sites. There are a number of complications; a) All the Gigabyte EP45 boards are slightly different, so while lifehacker.com might publish some nice howto’s for someone using a EP45-UD3P motherboard. My EP45-UD3LR is not ‘quite’ the same b) There are different ‘approaches’ you can take that have bizarre names that unless you’ve been trawling forums for quite some time you won’t understand; you can use hacked kexts, you can modify the DSDT, you can ‘inject’ EFI strings and seemingly you can use a combination of these .. though I’m not entirely sure. I think what’s happened is that the hackintosh community is obviously trying to increase the probability that when you do an Apple ‘Software Update’ that includes a kernel update (eg 10.6 to 10.6.1) that your machine doesn’t die painfully. I got the idea that modding the DSDT instead of injecting EFI strings somehow makes things ‘better’ and a big goal is to reduce the necessity for hacked kexts.

So does it work? Yes it does. A lot better than I thought it would even though I have had intermittent crashes (the screen goes dark and you get a message telling you to hold in the power button) which I think I’ve finally resolved (my Nvidia card seems to hate the 64 bit kernel drivers, but works OK booting in 32 bit mode). I won’t go into all the detail of what I did to build the thing, but here are my key points;

• I used the lifehacker.com ‘How to build a Hackintosh with Snow Leopard Start to Finish’ article (the original article, not the followup)
• I also referred to this EP45-UD3R guide on insanelymac. Especially the bits about setting the UUID in /Extra/smbios.plist and the PlatformUUID.kext. I found I couldn’t install Chameleon onto the hard drive until I got the UUID fixed
• I have an old GeForce 7300LE card with 128MB of RAM. This card is not listed in EFIStudio, but originally I just selected GeForce 7300 GT 256MB which seemed to work OK. I also noticed the Front Row would not work at all unless I had something like the following in /Extra/com.apple.Boot.plist:

      <key>Graphics Mode</key>
      <string>1280x1024x32</string>
  

• I used the ‘SL Pack’ that is referred to on various insanelymac howtos, especially the Extras folder it tells you to start with. ie. I didnt use the one that the LifeHacker article refers to. Though I did originally use the lifehacker one. I have no idea if there is any difference.
• To get my ALC888 sound to work required quite a few steps. There’s a really useful comment by Dith close to the bottom of this forum page. I followed the steps that start with ‘Redid my whole Snow Leopard installation’. And then I grabbed the LegacyHDA.kext that is at this path in the SL Pack and copied it into /Extra/Extensions replacing the one I already had.;

  SL Pack/DSDT Stuff/How to Patch DSDT /series of LegacyHDA 888 (ALC888)/3out2in HDA headphone/LegacyHDA.kext
  

• I manually enable QE/CI (well I think I did) by sudo’ing to root and running;

  defaults write /Library/Preferences/com.apple.windowserver GLCompositor -dict tileHeight -int 256 tileWidth -int 256
  

• I tended to run ‘Kext Utility’ between a lot of the above steps. Never sure if it makes much difference.
• I later found that OSX86Tools can generate the EFI string for my GeForce 7300LE 128MB card. So I used that in my /Extra/com.apple.Boot.plist hoping that it would fix my intermittent crash problem. It didn’t fix the crashes, but it did make me feel warm and fuzzy.
• I found that I could consistently crash the machine by running ‘Chess’ and just moving the mouse. Obviously Apple knows that I’m not very good at Chess, and was saving me some misery … These crashes were with a 64 bit kernel. Then I tried booting with a 32 bit kernel and Chess works fine, and so far no crashes (though I still can’t play Chess very well). I’m using Chameleon RC3 and had to put ‘arch=i386′ in the /Extra/com.apple.Boot.plist file like so;

          <key>Kernel Flags</key>
  	<string>arch=i386</string>
  

I’ve since updated from 10.6 to 10.6.1 with no dramas. The upgrade to 10.6.2 had some minor hitches in that you need to remove the older SleepEnabler.kext. See here for more info.

And of course, the new motherboard works fine running linux too!

It took me a while to get 2.6.31 to boot properly. I have an older nvidia card in this system, and the default setup had the kernel loading the nvidiafb module … which does not seem to work with my system (I just get a black or gray screen). Fortunately I could still ssh in from another box. It took me a while to work out how to properly disable nvidiafb, which ended up being somehting like;

cd /etc
mv modules.conf modules.conf.orig
cd modprobe.d
# Edit the blacklist file and add 'blacklist nvidiafb' to the end.
vi blacklist
# now update the initrd
update-initramfs -u

Also, in order to get X windows working I ended up downloading the driver from nvidia.com ( I used NVIDIA-Linux-x86_64-185.18.36-pkg2.run).

Once I had it booted (properly) off 2.6.31, I thought I’d have a look at btrfs. It’s a new filesystem for linux which is meant to be a lot like ZFS, but it’s not quite there yet in the stability department. However, I thought I read somewhere that the version included in 2.6.31 was ‘pretty good’, so it was worth a look. What I wanted to try was to have a btrfs root partition.

The first thing was to recompile my kernel again. Btrfs seemed to be turned off by default if you’re doing the make-kpkg stuff. So my setup process was something like;

cd /usr/src
tar xvjf /tmp/linux-2.6.31.tar.bz2
mv linux linux.orig
ln -s linux-2.6.31 linux
cd linux
make-kpkg clean
make menuconfig  <--- make sure you go to filesystems and enable btrfs as a module)
fakeroot make-kpkg --initrd --append-to-version=-custom2 kernel_image kernel_headers
# Wait 10 years
cd ..
dpkg -i linux-headers-2.6.31-custom2_2.6.31-custom2-10.00.Custom_amd64.deb
dpkg -i linux-image-2.6.31-custom2_2.6.31-custom2-10.00.Custom_amd64.deb

Now my plan was to make a copy of my root fs into a btrfs partition and boot off that. I have /boot as an ext2 partition, and my normal lenny root fs is on a logical volume as an ext3 fs. I thought I'd take a snapshot of the ext3 rootfs and copy it into a btrfs logical volume and boot off that. But the first thing to do is to get the btrfs user space tools;

git clone git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs-unstable.git
cd btrfs-progs-unstable
make
make install

Then setup the logical volumes and copy the data (NB: You might have to do a 'modprobe libcrc32c ; modprobe btrfs' first). For reference, I use a volume group called vgz which will undoubtedly be different to whatever system you might have.

mkdir /tmp/src
mkdir /tmp/dest
lvcreate -L 1G -s -n snap1 /dev/vgz/lvlenny64root
mount /dev/vgz/snap1 /tmp/src
lvcreate -L 12G -n lvlenny64btrfsroot /dev/vgz
mkfs.btrfs /dev/vgz/lvlenny64btrfsroot
mount -t btrfs /dev/vgz/lvlenny64btrfsroot /tmp/dest
cd /tmp/src
find . -print | cpio -dumpv /tmp/dest
cd /
# You'll need to edit the fstab to change the root device and filesystem type. For example:
#    /dev/mapper/vgz-lvlenny64btrfsroot /               btrfs    errors=remount-ro 0       1
vi /tmp/dest/etc/fstab
umount /tmp/dest
lvremove /dev/vgz/snap1
umount /tmp/src

Now we need to convince your initrd to load the btrfs modules (this is debian/ubuntu/whatever specific). I edited /etc/initramfs-tools/modules and added these lines in;

libcrc32c
zlib_deflate
crc32c
btrfs

And then you need to run update-initramfs -u
At this point I had to do some fiddling with /boot/grub/menu.lst. I just made an extra stanza in the file that looked like;

title           BTRFS Debian GNU/Linux, kernel 2.6.31-custom2
root            (hd1,1)
kernel          /vmlinuz-2.6.31-custom2 root=/dev/mapper/vgz-lvlenny64btrfsroot rootfstype=btrfs ro
initrd          /initrd.img-2.6.31-custom2

Obviously don't just copy this for your own setup, as the root line will be different and possibly some other stuff. Admitedly, I am probably missing something with this setup as when I go to boot off my btrfs root partition, I get kernel boot message up to the point where it's mounting the root filesystem, then just sits there for 5 minutes before eventually mounting it ok and continuing on. It's all a bit odd (UPDATE: This long pause is to do with the initramfs fstype and/or vol_id tools not being able to recognise btrfs filesystems. Eventually someone will update those tools, but until then you might want to edit /usr/share/initramfs-tools/scripts/local, hack the get_fstype function and then run update-initramfs-u. I'll put some notes at the bottom of the post).

Anyway, so I got a bootable system with btrfs. The first thing I wanted to try was snapshots, so I duely did something like;

btrfsctl -s snap1 /

When I first did that I thought it would create a snap1 directory under the / (ie. root) of my root fs. It didn't. It actually created a snap1 directory in the directory I was currently in. So within that directory was an intact snapshot of my root filesystem. Of course the first thing you want to do is delete the snapshot ... which in 2.6.31 is a 'future feature'. You can try and rm -rf the snapshot directory ... but I always got some circular file reference errors and it took ages as well. In fact I took a few snapshots as I played around, then left it running over night. In the morning it told me my filesystem was full and the hard drive light was permanantly on. Odd.

Fortunately, the ability to delete snapshots has actually been added to btrfs. You just need an even more recent kernel ... or what I did was to just check out the btrfs kernel tree (I went to here and clicked 'tree' up the top, then 'fs', then 'btrfs' then the 'snapshot' button near the top) and shove it into my 2.6.31 kernel and go through the whole make-kpkg thing again and try booting again (actually I did a few more steps and just deleted my whole btrfs example logical volume and created a new one with no snapshots).

Now with my '2.6.31 and a bit' kernel I can make a snapshot (I have now learned that putting a leading / in front of the snapshot name is probably sensible);

btrfsctl -s /snap1 /

And so if I cd /snap1 I can access my snapshot. But more importantly I can delete it;

btrfsctl -D snap1 /

The arguments to the 'btrfsctl -D' seem to be the snapshot name followed by the directory in which it was made.

I also found this post on root'ing the net that shows how you can mount the snapshot using the mount command (ie. in addition to the snapshot appearing as a subdirectory);

mount -o subvol=snap1 /dev/vgz/lvlenny64btrfsroot /mnt

Most stuff on btrfs indicates that subvols and snapshots are treated the same. If you create a new snapshot you get a copy of what you snapshotted. Creating a new subvol gives you an empty directory which I assume can be controlled separately, but uses the same pool of disk resources of my main btrfs filesystem. I'm not entirely sure. There is an option to the btrfsctl command which can resize a 'something' but I'm not too sure exactly what it does. I tried resizing down and df doesn't reflect any difference, but doing a btrfs-show did show some change. At the moment there aren't really any commands to list snapshots or show usage. There is a btrfs-debug-tree command that spits out reams of info ... and does mention my snapshots deep within it.

btrfs can apparently do RAID1, RAID0 and RAID10. I like that redundancy is a feature of the filesystem. I've never really understood why LVM2 for linux has no redundancy features (and the mirroring capability it does have is not very useful). I haven't tried any of the btrfs RAID features yet.

So that's all I've looked at so far. One of the key features of ZFS that I'd really like to see in btrfs is 'rollback to snapshot'. This is such an incredibly useful feature in a fast paced IT environment. Given that btrfs is a copy-on-write filesystem, I am hoping they put rollback in at some point.

Honestly I haven't tested it enough to determine how stable it is. It seems fine so far, but this is not a server type machine.

UPDATE: re getting rid of the long pause on boot.
I edited /usr/share/initramfs-tools/scripts/local, and added the extra lines shown below. Be warned, this is an awful 'hack' that assumes that any filesystem type that is not recognised is 'btrfs'. Once you've updated the file you'll need to run update-initramfs -u

get_fstype ()
{
        local FS FSTYPE FSSIZE RET
        FS="${1}"

        # vol_id has a more complete list of file systems,
        # but fstype is more robust
        eval $(fstype "${FS}" 2> /dev/null)
        if [ "$FSTYPE" = "unknown" ] && [ -x /lib/udev/vol_id ]; then
                FSTYPE=$(/lib/udev/vol_id -t "${FS}" 2> /dev/null)
        fi
        RET=$?

        if [ -z "${FSTYPE}" ]; then
                FSTYPE="unknown"
        fi

# Hack for my btrfs root. Terrible hack
        if [ "${FSTYPE}" = unknown ];then
                FSTYPE=btrfs
                RET=0
        fi
# End of my hack

        echo "${FSTYPE}"
        return ${RET}
}

I’ve used VNC for years, and it is indeed handy, but it’s never been a speed demon. Sure there are various compressions and encodings you can choose, but it’s never that great (however I have discovered tigervnc in the course of researching this article, and it does look very promising). I often end up running vnc in bgr233 (8 bit) mode in order to make the desktop more responsive, but at the expense of screwing up most of the colours on the screen. Of course, if you look at other remote screen technologies they tend to be a lot better. Windows terminal server (RDP) client access always seems quite fluid to me. Citrix always seems pretty good too, NoMachine is very good, but I always find it difficult to set up. The Sunray protocol is very good too.

So recently, I started looking at xrdp. It’s been around for ages, and almost looks like a dead project at times … but if you look closely at the mailing list … it is still ticking over. I always thought it was a RDP server for linux. Well it is sort of. Regardless of what it is, it provides you with a very responsive remote desktop experience.

So xrdp seems to do a few things. The main uses are as an authentication gateway to VNC or other RDP server. So if you run xrdp on your system it starts listening on port 3389 like a normal RDP server. You can connect to it with rdesktop or the Windows terminal services client. Basically you see a login screen. There’s a dropdown (yep the square grey box is a dropdown) that allows you to select different connection profiles. You define these yourself in /etc/xrdp/xrdp.ini. You always get a few examples in this file but they’re never really explained anywhere.

All the examples have a [xrdp] header. I’m not sure if they have to be [xrdp1], [xrdp2] etc. Below is an example of a profile that connects to a VNC server running on :1 (port 5901). name is what you see in the drop down. lib = libvnc.so means that you want xrdp to connect to a VNC server and then convert it back and forth between VNC and RDP. You need to start up your VNC server before hand though. You may think “Why not just use a VNC viewer and connect to the VNC server directly?”. Responsiveness. I find that accessing a typical linux desktop via this RDP to VNC method is actually a fair bit better than just connecting via VNC (and this is just on a local LAN). The other params are reasonably obvious. username=na means to not prompt for a username. password, ip and port are reasonably obvious. If they are set to ‘ask’ then you get to fill them in at the login screen. Obviously you can make this connect to any VNC server by using password=ask, ip=ask, port=ask

[xrdp1]
name=VNC5901
lib=libvnc.so
username=na
password=ask
ip=127.0.0.1
port=5901

This next example is more interesting;

[xrdp4]
name=sesman-any
lib=libvnc.so
ip=ask
port=-1
username=ask
password=ask

It still uses libvnc.so so it’s obviously converting between RDP and VNC still. The interesting bit is the username=ask. Most VNC servers don’t take a username. What this profile does is to prompt for a username and password and ip. As part of the xrdp startup it always starts up a daemon called sesman. This is like a login authenication daemon which can spawn a command once authenticated. By default sesman listens on port 3350, so xrdp takes your user/pass and connects to the ip you specified on port 3350. The sesman on that port should respond and authenticate the user (typically the ip is always 127.0.0.1). Then the cool bit is that it starts up Xvnc as your user on the ‘next available DISPLAY setting above :10′, and assuming the Xvnc startup works, xrdp starts the whole RDP to VNC conversion. So you no longer need to start up Xvnc beforehand. Another good thing about this method is that if you just exit your RDP client (ie. don’t choose to logout), then the Xvnc session keeps running in the background, and if you RDP in again later it’s all still working where you left off. Conversely, if you choose to ‘log out’ then the Xvnc process and any windows you had open ends as you log out.

But there are more lib= types you can use. Look at this example from xrdp.ini that uses lib=librdp.so

[xrdp5]
name=windows
lib=librdp.so
ip=ask
port=3389

librdp.so allows you to sort of proxy through your xrdp server to another RDP server (eg. a windows PC)

Then there’s this sesman-X11rdp example;

[xrdp6]
name=sesman-X11rdp
lib=libxup.so
username=ask
password=ask
ip=127.0.0.1
port=-1

For a long time I never really understood what this one did. In fact in 99% of all xrdp installs out there it does absolutely nothing. It asks for your username so that means that sesman thing is involved. So it will authenticate your user/pass with sesman, but instead of starting Xvnc, it starts X11rdp. Chances are you do not have X11rdp. Most linux distros do not include it as part of their ‘xrdp’ packages, so what is it?

X11rdp is a real X server that instead of thinking its talking to say an Nvidia or ATI hardware video card, it is talking to an abstracted ‘RDP’ video card. Similarly the mouse and keyboard are abstracted to come down the RDP protocol connection. So it still talks the X protocol, so that typical X client programs (eg. xterm) still think they’re talking to an X server, but ‘out the other side’ it’s all RDP traffic. I don’t think it talks RDP over a network connection though (not sure). I think the role of libxup.so is to provide the ‘link’ to it.

So where do you get X11rdp? I couldn’t find many references to it on the net but came across this post that mentions that there is an svn repository containing a modified Xorg 7.1 server that when compiled produces the X11rdp binary. Compiling the old Xorg drivers takes a long long time, but I had success following those notes. So here’s what I did;

cd some_temp_dir_with_plenty_of_space
svn co svn://server1.xrdp.org/srv/svn/repos/main/x11rdp_xorg71
cd x11rdp_xord71
mkdir /opt/X11rdp
sh buildx.sh /opt/X11rdp
# and now wait a long long time

That post on linuxquestions.org mentioned some changes to font directory paths inside the buildx.sh script. I didn’t have to change it at all the first time I tried compiling. I used a Debian Lenny 64 bit host to compile. However, I wanted to put this on a Centos 5.3 32 bit system as well and it wasn’t so good. basically there are a couple of changes to the buildx.sh script. First it compiled libfreetype as a 64 bit binary, so I had to add in the CC=”gcc -m32″ bit below

# freetype
if ! test -f $PCFILEDIR/freetype2.pc
then
  cd freetype-2.1.10
  CC="gcc -m32" ./configure --prefix=$PREFIXDIR
  if ! test $? -eq 0
  then
    echo "error freetype"

And the font path in Centos is a bit different to what the script was expecting, so I had to do the following (adding in the last elif bit)

# make a symbolic link to your local font directory
if ! test -d $X11RDPBASE/lib/X11/fonts
then
  if test -d /usr/share/fonts/X11
  then
    ln -s /usr/share/fonts/X11 $X11RDPBASE/lib/X11/fonts
  elif test -d /usr/X11R6/lib/X11/fonts
  then
    ln -s /usr/X11R6/lib/X11/fonts $X11RDPBASE/lib/X11/fonts
  elif test -d /usr/share/X11/fonts
  then
    ln -s /usr/share/X11/fonts $X11RDPBASE/lib/X11/fonts
  fi
fi

I also compiled for a Debian 32 bit build and had to put ‘CC=”gcc -m32″;export CC’ at the very top of the build script (but that might be because of the virtual machine I was compiling in).

So once you have X11rdp compiled, make sure it’s in your path (perhaps create a symlink from /usr/bin/X11rdp to wherever you have it).

So for me xrdp plus X11rdp works great, but you could never copy and paste between your local environment and your remote environment … until now. As well as the stable releases of xrdp, there is also a CVS repository, and one of the more recent commits (sept 2009) is to add clipboard support (thank you Jay). To get and compile the cvs version you need to check it out.

mkdir workdir
cd workdir
cvs -d:pserver:anonymous@xrdp.cvs.sourceforge.net:/cvsroot/xrdp checkout xrdp
cd xrdp
./bootstrap
./configure
make install

You might have to do some fiddling with various paths to get it all going (most distro’s seem to run xrdp as a non-root user. If you run the cvs version this way it will have trouble writing the pid file when xrdp starts … unless you change the path to the pid file … or do a ./configure –prefix=/usr/local/xrdp … or similar. The best way is to just use strace when launching the binaries to see what’s going on)

You run xrdp then xrdp-sesman (which used to be called just ’sesman’). The easy way is to run both as root, though you might want to invest the time to get xrdp to run as a non-root user. For example;

su - xrdp -c /usr/local/sbin/xrdp
/usr/local/sbin/xrdp-sesman

The clipboard support just ‘works’ for the most part. I had a few issues with using the Windows XP mstsc client, but rdesktop and the Windows 7 mstsc seem to work fine (UPDATE: It’s only the XP SP2 mstsc that I can’t get to work with the clipboard. The mstsc out of XP SP3 works fine … and in fact works fine if you copy it to a SP2 machine)

ESX/ESXi 4 is quite different from 3.5. The build chain is similar to 64 bit Redhat/Centos 5.2, so I ended up installing a x86_64 Centos 5.3 inside a vmware fusion machine to do my dev work. I just made sure I installed all the dev stuff.  Then I downloaded the VMware-esx-public-source-4.0-162945.tar.gz from VMware’s open source page. It’s a much bigger file (590MB) than the file for 3.5. When you extract the file you end up with a lot of rpm files plus a vmkdrivers-gpl.tgz file. I did the following to extract it all on my test machine;

cd ~
mkdir vmware-oss
cd vmware-oss
tar xvzf ~/VMware-esx-public-source-4.0-162945.tar.gz
mkdir drivers
cd drivers
tar xvzf ../vmkdrivers-gpl.tgz

One of the rpm files included is a kernel source rpm. I’m not exactly sure what it is relevant to ESX, but I installed it anyway for reference. I found I needed the qt-devel and gtk2-devel packages first;

cd ~
cd vmware-oss
yum install qt-devel
yum install gtk2-devel
rpm -iv kernel-sourcecode-400.2.6.18-128.1.1.0.4.159770.x86_64.rpm

I’m pretty sure you don’t need the kernel source to build the drivers, but I kept it handy for reference anyway.

You can try doing a test build of the drivers now. This will build all the drivers built in to ESX/ESXi.

cd ~/vmware-oss/drivers
./build-vmkdrivers.sh

You’ll probably get a few warnings, but it should complete. If you do a find down the ‘bora’ directory you should see a bunch of .o files corresponding to the kernel modules (look under the ‘bora/build/scons/build’ directory).

OK, my approach was to look at the build-vmkdrivers.sh script and basically look at what was done to compile one network driver (I used the forcedeth driver as a reference) and just make a reduced script for my sky2 driver.  As for the source to base the sky2 driver on, instead of using a driver from 2.4.37 like I did with the 3.5 version of the network driver, this time I ended up using the sky2 source from 2.6.26 (or the debian lenny incantation of it). I did originally use the sky2 driver from the kernel-sourcecode…2.6.18…rpm file, but on closer inspection of the tg3 driver that ESX uses, I noticed it actually comes from a 2.6.24.1 kernel (or thereabouts) … so I thought I may as well use a more modern reference source. I had a few minor hiccups trying to get it to compile, but in the end I just had the following shoved into the top of my sky2.c file;

/* Stuff for ESX compile */
#define upper_32_bits(n) ((u32)(((n) >> 16) >> 16))
#define csum_offset csum
#define bool int
#define PTR_ALIGN(p, a)         ((typeof(p))ALIGN((unsigned long)(p), (a)))

u32 bitreverse(u32 x)
{
        x = (x >> 16) | (x << 16);
        x = (x >> 8 & 0x00ff00ff) | (x << 8 & 0xff00ff00);
        x = (x >> 4 & 0x0f0f0f0f) | (x << 4 & 0xf0f0f0f0);
        x = (x >> 2 & 0x33333333) | (x << 2 & 0xcccccccc);
        x = (x >> 1 & 0x55555555) | (x << 1 & 0xaaaaaaaa);
        return x;
}

The define's are to remedy compilation errors, and the bitreverse is to satisfy an undefined symbol problem. Note that the undefined symbols errors end up in /var/log/messages on your ESXi 4 box now (unlike 3.5).

So in the end to compile my driver I did a;

./build-sky2.sh

You get a couple of warnings, but if you do a 'find . -name sky2.o' you should end up with two sky2.o files. There is a DEBUG and DASHG variable defined at the top of the build script. If you uncomment these it'll build a lot of debug stuff into the modules.

If you get some errors, its probably because some directories are missing in the build path, so make them first;

mkdir -p bora/build/scons/build/vmkdriver-sky2.o/release/vmkernel64/SUBDIRS/vmkdrivers/src26/drivers/net/sky2
mkdir -p bora/build/scons/build/vmkdriver-sky2.o/release/vmkernel64/SUBDIRS/vmkdrivers/src26/common/

Now, again I used a USB stick with ESXi 4 (build 171294 in my case). To install it, I loopback mounted the VMware ESXi 4 iso file file, extracted the image.tgz file to a temp directory, bunzip2'd the big dd image file, then dd'd it to the whole USB stick.

A difference this time is that I didn't have a simple.map file all pre-prepared to go into the oem.tgz file. I thought the easiest way to get it would be to just boot ESXi and let it fail when it tries to configure a network device, then somehow copy the simple.map file off. So I did this. ESXi 4 merrily boots and eventually you see the dreaded 'lvmdriver failed' message. It looks like ESXi is broken at that point, but just type the word 'unsupported' and you get a password prompt, and just hit ENTER to get a prompt.

Because networking is not working, we'll just copy the simple.map to the Hypervisor1 partition;

cp /etc/vmware/simple.map /vmfs/volumes/Hypervisor1

I just did a 'sync' and held in  the power switch ( perhaps type 'reboot' if you feel like being more careful). Now get the USB stick to appear as a USB device in your development VM (your centos 5.x environment), and mount partition 5 (or the Hypervisor1 partition) off the USB drive, and you should see an oem.tgz file as well as the simple.map file. You need to make a directory structure up for the new oem.tgz file we'll be creating;

cd ~
mkdir vmtest
cd vmtest
mkdir -p etc/vmware
mkdir -p usr/lib/vmware/vmkmod

Copy the simple.map off the USB drive into etc/vmware directory in our tree structure. eg.

cp /mnt/simple.map ~/vmtest/etc/vmware

And edit the simple.map so that it includes the PCI ids for your Marvell card. Mine is 11ab:4362 so I added in the bolded line below, but yours could likely be different. If you're not sure, you could boot ESXi off the USB stick again, do the 'unsupported' thing to get a prompt and type lspci -v

1166:0410 0000:0000 storage sata_svw.o
1166:0411 0000:0000 storage sata_svw.o
11ab:4362 0000:0000 network sky2.o
14e4:1600 0000:0000 network tg3.o

Now copy in the sky2.o file that we compiled earlier. The modules are in a different directory compared to 3.5 (NB: the compilation process produces two sky2.o files, so make sure you grab the one shown below)

cd ~/vmware-oss/drivers
cp ./bora/build/scons/build/vmkdriver-sky2.o/release/vmkernel64/sky2.o ~/vmtest/usr/lib/vmware/vmkmod

Now tar it up, and copy it to the USB stick that should be still mounted;

cd ~/vmtest
tar cvzf ~/oem.tgz *
cp ../oem.tgz /mnt

Unmount the USB stick

umount /mnt

Now try booting again. Hopefully you should see a 'loading sky2' flash up early in the boot ... and it should eventually get to the usual ESX status screen showing the current mgmt IP address. Basically if you don't see the 'lvmdriver load failed' then there's a good chance it's working.

And yes here is the sky2-for-esxi4-0.01.tar.gz download. It includes the build script, the modified source, plus directory tree for creating the oem.tgz file including a pre-compiled copy of the module. If you can't be bothered compiling, you can just extract this file, cd to the vmtest directory and create the oem.tgz file as per the earlier notes.

UPDATE: (2010/02/08) There is also now a driver for the Marvell 88E8001 LAN chipset (see the comments discussion below). This uses the skge driver, not the sky2 driver mentioned above. I don't own a 88E8001, so thank you to samarium for helping out re the unresolved symbols. Please comment below if you've tried the skge driver and it works. I have a new tarball sky2-and-skge-for-esxi4-0.02.tar.gz containing both the sky2 and skge driver

However, there are a few projects on the net to port drivers to ESXi since the driver layer is based on some older linux kernels as far as I can tell. I found the open-vdrivers project on sourceforge that has some modules for some Realtek gigabit LAN cards for using with ESXi 3.5. There is also some info on the open-vdrivers site re setting up the build environment to compile the  drivers  for ESXi 3.5. Apparently you need an old gcc compiler (3.2 or 3.3 I think), and the recommendation was to use Centos 3 as a platform. It appeared as though they had taken the realtek driver source from a 2.6.18 kernel and modified it to get it to go in the VMware build environment.

So I thought I’d attempt to port the linux sky2 driver (what the Marvell Yukon2 LAN chipsets use) compiled and running on ESXi 3.5.

How hard could it be? (I must admit I’m a hackerish C programmer at the best of times and didn’t have high hopes).

Amazingly I got it to work. I honestly don’t know how well it works, but I’ve performed backups over the network with it and it hasn’t died.

So I installed Centos 3.9 i386 in a virtual machine, ticking the boxes to make sure I had the dev environment stuff installed (I ended up needing all three install CDs). I also needed subversion to check out the open-vdrivers project and I must admit trying to find subversion for Centos 3 was painful (I think I ended up finding a Subversion 1.2 rpm somewhere that worked). It’s probably easier to check out open-vdrivers on a more recent distro and then just copy the directory tree into your Centos 3 VM.

The notes on setting up the build enviornment for open-vdrivers is here at the moment. I’ll repeat a fair chunk of this below. You need the open sourced files from Vmware to get started. I was using 3.5u3 initially (since that was the only install iso I had), but later repeated the process with 3.5u4. All the notes below will just refer to 3.5u4. And yes, yes, I know 4i is out, but I haven’t even attempted to look at 4i yet.

So go to the VMware open source page and download the 3.5u4 OSS source code. The file I ended up with was VMware-eesx-public-source-3.5.0-153875.tar.gz. Extract that file in your Centos3 machine and you should end up with a bunch of other tar.gz files;

VMware-eesx-gpl-public-source-3.5.0-153875.tar.gz
VMware-esx-server-public-source-3.5.0-153875.tar.gz
VMware-esx-drivers-public-source-3.5.0-153875.tar.gz

Extract all those in a temporary directory. You’ll end up with a few more tgz files which aren’t that important as well as two directories;

VMware-esx-drivers-public-source-3.5.0-153875
VMware-esx-public-source-3.5.0-153875

Check out the open-vdrivers project inside the same temp dir (or check it out on another machine and then transfer the ‘open-vdrivers’ directory to your centos3 machine.

svn co https://open-vdrivers.svn.sourceforge.net/svnroot/open-vdrivers/trunk open-vdrivers

First you have to add a symlink into the VMware source distribution. In my case;

cd VMware-esx-drivers-public-source-3.5.0-153875/src/include
ln -s ../../../VMware-esx-public-source-3.5.0-153875/include/lkhdrs/asm-i386

Now cd into the VMware-esx-drivers-public-source-3.5.0-153875 directory inside your temp directory and execute the build script for VMware’s own open sourced drivers;

cd ~/tmp/VMware-esx-drivers-public-source-3.5.0-153875
./build-vmkdrivers.sh

You’ll get a fair few warnings and mine always ends in an errors about the usb-storage driver (see below). I don’t really care about that, and I don’t think it really matters for what you’re about to do.

**** Creating library drivers-usb-storage-obj.a
ar: build/release/drivers/usb/storage/debug.o: No such file or directory
ld: cannot open build/release/drivers/usb/storage/bin/drivers-usb-storage-obj.a: No such file or directory

Now we need to do a test compile of the open-vdrivers code. First you need to set up another symlink;

cd ~/tmp/open-vdrivers/src
ln -s ../../VMware-esx-drivers-public-source-3.5.0-153875/src/include

And then try compiling one of the realtek drivers to check that it succeeds;

./build-r8169.sh
**   Compiling        drivers/net/r8169/r8169.c
**   Compiling        drivers/net/r8169/linux_module_heap.c
**** Creating library drivers-net-r8169-obj.a

The actual module is in  build/release/drivers/net/r8169/bin/

ls -l  build/release/drivers/net/r8169/bin/r8169.o
-rw-r--r--    1 root     root       979049 Aug 14 23:01 build/release/drivers/net/r8169/bin/r8169.o

OK, so far so good. Now, if you just want to try compiling my sky2 build, grab my sky2-for-esxi35u4-0.01.tar.gz and extract it into the open-vdrivers directory, then compile it;

./build-sky2.sh

And then you should end up with the driver in build/release/drivers/net/sky2/bin;

-rw-r--r--    1 root     root       975909 Aug 14 23:15 sky2.o

Yeah, I’m not so sure why its that big either. To get it to work on ESXi, I find it much easier if ESXi is on a USB stick. There’s a few howtos on the net for how to do this. Most seem to be Windows-centric. Here’s my brief linux take on it;

cd /tmp
mkdir vmiso
mount -o loop VMware-VMvisor-InstallerCD-3.5.0_Update_4-153875.i386.iso /tmp/vmiso
mkdir /tmp/install
cd /tmp/install
tar xvzf /tmp/vmiso/install.tgz
cd usr/lib/vmware/installer
# Insert a USB stick of at least 1GB in size. Make sure it doesn't mount. If your linux distro auto
# mounts it then unmount it. Run dmesg to find out what its device name is. In my case it was sdd,
# so I put of=/dev/sdd below. This will vary depending on your machine. MAKE SURE YOU GET IT
# RIGHT, otherwise you'll trash one of your harddisks)
bunzip2 -c VMware-VMvisor-big-3.5.0_Update_4-153875.i386.dd.bz2 | dd of=/dev/sdd bs=32k

That bunzip2/dd thing will take a while. Wait until your USB stick activity LED stops flashing and then probably pull it out and stick it back in again. If you do a dmesg again you might see some partiion info;

[560078.250013]  sdd: sdd1 < sdd5 sdd6 sdd7 sdd8 > sdd4

So that gets you a default install. Now how do we insert the sky2 driver into it? You need to firstly get the sky2.o driver onto the USB stick and then you need to make sure the PCI id information is updated so that it loads the correct driver at boot time. If you’re currently running linux on the machine you want to use for ESXi, you can run lspci to work out the PCI id for your card. On mine;

# lspci -nn

...
02:00.0 Ethernet controller [0200]: Marvell Technology Group Ltd. 88E8053 PCI-E Gigabit Ethernet Controller [11ab:4362] (rev 19)
...

The 11ab:4362 bit is my PCI id. You need to write this down.

There are few possibilities for which tar.gz file to change on the USB stick, but I used the oem.tgz method. In my case I had already downloaded the Unified oem.tgz from acronymlabs . This oem.tgz is actually for ESXi 3.5u3, but it works well enough for me on u4.  Extract that oem.tgz file into a temp directory on your linux box,

cd /tmp
mkdir oemstuff
cd oemstuff
tar xvzf ~/CommunityUnifiedDriverPack_v1.1.0_U3-123629.oem.tgz

then edit the etc/vmware/simple.map file and add an entry in for your PCI id. In my case, I added the bolded entry below;

1166:024b 0000:0000 storage sata_svw
1166:0411 0000:0000 storage ide
11ab:4362 0000:0000 network sky2
13c1:1003 0000:0000 storage 3w_9690

Now copy the build/release/drivers/net/sky2/bin/sky2.o file from your build directory into the mod directory

And create a new tar.gz file and make sure its called oem.tgz

cd /tmp/oemstuff
tar cvzf /tmp/oem.tgz *

Insert the USB stick and mount partition 5 (or the one with the label Hypervisor1)

mkdir /tmp/usb
mount LABEL=Hypervisor1 /tmp/mnt

You’ll see that there already should be a tiny oem.tgz file in that partition. You can back it up if you like, but there is pretty much nothing in it. So just copy your new oem.tgz file to the USB stick

cp /tmp/oem.tgz /tmp/mnt

And unmount it

umount /tmp/mnt

Now try booting off the USB stick (you might need to change your BIOS settings for USB booting, or I ended up pressing F8 to get the boot select menu most of the time). I disconnected all my linux hard disks before rebooting as a precaution. You can boot off the USB stick with no physical hard drives in your system, and on this first boot we just want to confirm that the sky2 card is recognised.

So you should see the ESXi boot up messages and some progress info about loading drivers. If you see ‘lvmdriver failed to load’, then that usually means your network card driver didn’t load. Once all the boot up stuff has finished you should get the status screen with an IP address onscreen (probably 0.0.0.0 initially). Now you can press Alt-F1 and type the word ‘unsupported’  (you won’t see any of these letters as you type it). Then you’ll get a root password prompt. just hit ENTER as you probably haven’t assigned a root password yet. Now you should have # root prompt. Have a look at the /var/log/config.log file;

less /var/log/config.log

Scroll through the file, and find where its loading the sky2 driver.Hopefully you should see something like;

Using /mod/sky2.o
Module load of sky2 succeeded

If you don’t, or you see messages about unresolved symbols then it hasn’t worked. If it has worked, then you can ‘exit’ and press Alt-F2 to go back to the main status screen. Press F2 to configure ESXi. Assuming the network card is working, you should be able to configure an IP address for the mgmt interface as well as a hostname. Go to a machine running Windows, overcome your depression, and point a browser at https://<the IP you assigned to your ESXi box> and you’ll get a cert warning and then a default page that allows you to download the Infrastructure client from ESXi itself. Install that and configure your ESXi box.

For reference, I started out using the sky2 driver from a 2.6.18 kernel (since the realtek drivers seem to use this version as a base), but I wasn’t making a lot of progress, so then I read that ESXi 3.5 is closer to a 2.4.21 kernel, so I used the most recent 2.4 kernel (2.4.37) and grabbed the sky2 driver from it. That turned out to be a lot quicker to get to the point where it actually compiled successfully (NB: the final ar in the build script will always fail unless the ‘bin’ directory exists for the output file). Once I had it successfully compiled, it still failed to work in ESXi. Looking at the /var/log/config.log I was getting a few unresolved symbols alerts when it tried to load the sky2 module. I noted down the names of these symbols, then went back to my centos3 build system and searched a vanilla 2.4.37 kernel tree looking for these missing symbols. Fortunately they tended to be simple functions that had no further dependencies, so I basically just copied the code from the kernel directly into the driver and recompiled, put it all on the USB stick again and rebooted until the module loaded cleanly … and then amazingly it worked. I also tried hacking at the 3c59x driver in order to get a 3com 3c905 boomerang card to work (which seems to be the one type of 3c905 that is not supported by ESXi). Again I got it to compile and load, but honestly I don’t think it works properly. I had it working sort of OK as a secondary interface, but if ESXi decided to use it as vmnic0 (ie. the mgmt interface nic) then ESXi would crash on boot.

And finally a big disclaimer. I am certainly not a kernel driver programmer by any stretch. I somehow got this driver to compile and amazingly it works fine on my home network in the limited testing I’ve done. I still think for production use, go buy some intel pro 1000 LAN cards or get a motherboard with an intel gigabit lan integrated.

UPDATE: 2010/03/27. On my ‘Using a Marvell LAN card with Vmware ESXi 4′ post there’s been some discussion about using a different type of Marvell card; a 88E8001 which uses the linux skge driver. I’ve finally had some feedback that my latest attempt at an skge driver might be working. Thats under ESXi4. So I’ve had an attempt at doing much the same hackery of the linux 2.4.x skge drive for ESXi3.5 . So if you have a 88E8001 and want to try it with ESXi 3.5, grab skge-for-esxi3.5u4-0.04.tar.gz and post some feedback as to whether it works or not. You’ll need to edit the simple.map file and add an entry like;

11ab:4320 0000:0000 network skge.o

If the module does not work, please post any ‘unresolved symbol’ info you find in the logs. As per the other post, I don’t own an 88E8001 based device so cannot test it myself. This v0.04 release adds some suggestions by ‘Sheepless’ (see comments), but I am dubious that it will actually work. As per Sheepless’ comments, this skge driver might be a bit harder to do for ESXi 3.5. But, if you’re able to try it, please do, and post some feedback.

So yes the keyboard still worked, but it looked pretty awful. I’m often at client sites with the laptop and it is slightly embarrassing to turn up with a broken looking laptop. So off to ebay we go, and I bought yet another replacement keyboard. Three keyboards in 12 months is pretty sad. I quite like OSX as a platform, but apple hardware quality is definitely debatable.

Now, the original keyboard on my macbook is what is often called a V3 keyboard (at least that’s what a lot of ebay sellers call it). It has the mute/softer/louder buttons over F10, F11 and F12. A V2 keyboard has the mute/softer/louder over the F3, F4 and F5 keys. There’s a few other differences as well. My 2nd keyboard (the one now with the crack in it) was a V2 keyboard. It was in good condition when I bought it. I probably prefer the layout of the V2 keyboard over the V3, but this time I ended up buying another V3 keyboard on ebay. It was a factory second with some scuff marks on it that you can only see if you angle the keyboard in the light a certain way. My thoughts are that its effectively brand new, so hopefully it will last more than 10 months.

I’ve been using rsync for quite some time to backup directories between linux and other types of Unix systems. It works great. It’s often as simple as;

rsync -av -e ssh source_dir user@remotehost:destdir

Little did I know, rsync to/from a Mac to a non-Mac unix is actually quite complicated. Unlike most other Unix filesystems, the HFS+ filesystem that Macs use includes a bunch of extra ‘metadata’ such as resource forks, finder-flags, finder-locks, a special type of creation-date, bsd-flags etc. Obviously, if you’re copying a Mac directory elsewhere for backup purposes you’d like to be able to restore all your files plus this ‘extra metadata’.

Apparently the oldish version of rsync (v2.6.9 on my OSX 10.5.7) has the Apple specific -E special option to allow it to capture this extra ‘metadata’, but it only really works if you’re copying to another directory on the same server, or to another Mac running the same modded Apple rsync. Apparently, the more recent versions of rsync (latest is 3.0.6) include features that allow them to save/restore most of the OSX specific ‘metadata’.

I ended up installing the latest version of rsync on my macbook using macports. Interestingly, it is v3.0.5 plus some additional patches for OSX (fileflags and crtimes patches). It installs under /opt/local/bin. I also installed backup-bouncer which can create an OSX volume containing a set of test files, each with some of the more unusual features of OSX. You then use your directory copying program to copy the test files elsewhere and then use backup-bouncer to check if all the OSX metadata were copied across.

Using the macports rsync, it is really good at rsyncing between two local directories or to a directory on another Mac. For example, if I do the following, backup-bouncer’s verify tells me my ‘testdir’ passes all tests;

sudo /opt/local/bin/rsync -av --crtimes --hard-links --acls --xattrs --fileflags /Volumes/Src testdir

However, I can’t do that when transferring to a non-mac. rsync generally requires that the version of rsync at the destination end also understands your command line arguments. I attempted to compile rsync on my Debian Lenny server using the same build technique the macports version uses (ie. with the crtimes and fileflags patches). This does not work, since those two extra patches require a bunch of OSX specific header files. So that left me with compiling the normal rsync 3.0.5 from http://www.samba.org/rsync/download.html. That compiled fine on my Debian box, and I ended up copying it over the /usr/bin/rsync that I already had on the server.

So I couldn’t use the crtimes or fileflags patches. Backup-bouncer doesn’t class those things as ‘critical’, so I thought from a backup point of view, I’d leave them out. So I tried this to copy data to the linux box ‘host’

/opt/local/bin/rsync -av --hard-links --acls --xattrs -e ssh /Volumes/Src user@host:/backup

That also did not work. The /backup filesystem on the linux host was ext3, but you need to use additional mount options if you want to use extended attributes or ACLs, so I remounted my backup partition;

umount /backup
mount -t ext3 -o user_xattr,acl /dev/hdc1 /backup

Now, it works a bit better, but a lot of stuff still fails. The basic problem is that there are still a bunch of OSX specific metadata bits that rsync on the Mac can read, but the linux end does not know how to store them. For this, rsync has the –fake-super option. It basically encodes the extra stuff into the extended attributes on the remote system.

I finally ended up with the following that MOSTLY works. It still fails on some of the non-critical backup-bouncer tests. It also cannot handle large amounts of metadata

• Back up a directory from the Mac using;

  /opt/local/bin/rsync -aHv --acls --xattrs -e ssh  \
  --rsync-path="rsync --fake-super"  source_dir user@linuxhost:/backup/latest

• Restore a directory using.

  /opt/local/bin/rsync -aHv --acls --xattrs -e ssh \
  --rsync-path="rsync --fake-super"   user@linuxhost:/backup/latest restore_dir

One thing I have read is that the size that ext3 allows for extended attributes is quite small, and if you have a lot of metadata for a file, then this small space can fill up. I’ve noticed this with backups of photos. The resource fork info for each photo is quite substantial and I get alerts like this when I run rsync;

rsync: rsync_xal_set: lsetxattr("Pictures/iPhoto Library/Originals/2008/8:11:2008/IMG_0023.JPG",
"user.com.apple.ResourceFork") failed: No space left on device (28)

On OSX, you can examine attributes with the OSX xattr command;

$ cd 'Pictures/iPhoto Library/Originals/2008/8:11:2008/'

$ xattr IMG_0023.JPG
com.apple.FinderInfo
com.apple.ResourceFork
com.apple.metadata:kMDItemSupportFileType

$ xattr -pl com.apple.ResourceFork IMG_0023.JPG

When you enter that last command you end up with a hex dump of the resourcefork thing. My output ends with B0B0 as the last line … which is about 45Kbytes. I think ext3 has a limit of 3.9K for all extended attributes, hence the error. I found this post that suggested using XFS as it can handle much larger extended attribute data (maybe 64K).

I tried XFS, but ultimately gave up on it, since I always had bad delete performance. Now, I’ve settled on reiserfs. It can also handled a large extended attribute set (NB: You have to specify acl,user_xattr with reiserfs, just like the examples for ext3). I still get a few errors with my backups re issues with backing up metadata, but it works well enough for my purposes.