KernelCrash

There are quite a lot of ways to get a remote graphical desktop when connecting to a unix system. X windows itself has always been a network based protocol, so if you had an X terminal or a PC running an X server, or even another unix system, you could always connect in that way. However, in more modern times with locked down Windows desktops in corporations, VNC has become very popular. Since something like the tightvnc client is just a standalone executable (and doesnt require installation), it’s often the simplest way of getting a graphical desktop going from a Windows PC. Another advantage is that VNC just uses a single tcp port connection, so it’s often a simple port forward and you can connect.

I’ve used VNC for years, and it is indeed handy, but it’s never been a speed demon. Sure there are various compressions and encodings you can choose, but it’s never that great (however I have discovered tigervnc in the course of researching this article, and it does look very promising). I often end up running vnc in bgr233 (8 bit) mode in order to make the desktop more responsive, but at the expense of screwing up most of the colours on the screen. Of course, if you look at other remote screen technologies they tend to be a lot better. Windows terminal server (RDP) client access always seems quite fluid to me. Citrix always seems pretty good too, NoMachine is very good, but I always find it difficult to set up. The Sunray protocol is very good too.

So recently, I started looking at xrdp. It’s been around for ages, and almost looks like a dead project at times … but if you look closely at the mailing list … it is still ticking over. I always thought it was a RDP server for linux. Well it is sort of. Regardless of what it is, it provides you with a very responsive remote desktop experience.

So xrdp seems to do a few things. The main uses are as an authentication gateway to VNC or other RDP server. So if you run xrdp on your system it starts listening on port 3389 like a normal RDP server. You can connect to it with rdesktop or the Windows terminal services client. Basically you see a login screen. There’s a dropdown (yep the square grey box is a dropdown) that allows you to select different connection profiles. You define these yourself in /etc/xrdp/xrdp.ini. You always get a few examples in this file but they’re never really explained anywhere.

All the examples have a [xrdp] header. I’m not sure if they have to be [xrdp1], [xrdp2] etc. Below is an example of a profile that connects to a VNC server running on :1 (port 5901). name is what you see in the drop down. lib = libvnc.so means that you want xrdp to connect to a VNC server and then convert it back and forth between VNC and RDP. You need to start up your VNC server before hand though. You may think “Why not just use a VNC viewer and connect to the VNC server directly?”. Responsiveness. I find that accessing a typical linux desktop via this RDP to VNC method is actually a fair bit better than just connecting via VNC (and this is just on a local LAN). The other params are reasonably obvious. username=na means to not prompt for a username. password, ip and port are reasonably obvious. If they are set to ‘ask’ then you get to fill them in at the login screen. Obviously you can make this connect to any VNC server by using password=ask, ip=ask, port=ask

[xrdp1]
name=VNC5901
lib=libvnc.so
username=na
password=ask
ip=127.0.0.1
port=5901

This next example is more interesting;

[xrdp4]
name=sesman-any
lib=libvnc.so
ip=ask
port=-1
username=ask
password=ask

It still uses libvnc.so so it’s obviously converting between RDP and VNC still. The interesting bit is the username=ask. Most VNC servers don’t take a username. What this profile does is to prompt for a username and password and ip. As part of the xrdp startup it always starts up a daemon called sesman. This is like a login authenication daemon which can spawn a command once authenticated. By default sesman listens on port 3350, so xrdp takes your user/pass and connects to the ip you specified on port 3350. The sesman on that port should respond and authenticate the user (typically the ip is always 127.0.0.1). Then the cool bit is that it starts up Xvnc as your user on the ‘next available DISPLAY setting above :10′, and assuming the Xvnc startup works, xrdp starts the whole RDP to VNC conversion. So you no longer need to start up Xvnc beforehand. Another good thing about this method is that if you just exit your RDP client (ie. don’t choose to logout), then the Xvnc session keeps running in the background, and if you RDP in again later it’s all still working where you left off. Conversely, if you choose to ‘log out’ then the Xvnc process and any windows you had open ends as you log out.

But there are more lib= types you can use. Look at this example from xrdp.ini that uses lib=librdp.so

[xrdp5]
name=windows
lib=librdp.so
ip=ask
port=3389

librdp.so allows you to sort of proxy through your xrdp server to another RDP server (eg. a windows PC)

Then there’s this sesman-X11rdp example;

[xrdp6]
name=sesman-X11rdp
lib=libxup.so
username=ask
password=ask
ip=127.0.0.1
port=-1

For a long time I never really understood what this one did. In fact in 99% of all xrdp installs out there it does absolutely nothing. It asks for your username so that means that sesman thing is involved. So it will authenticate your user/pass with sesman, but instead of starting Xvnc, it starts X11rdp. Chances are you do not have X11rdp. Most linux distros do not include it as part of their ‘xrdp’ packages, so what is it?

X11rdp is a real X server that instead of thinking its talking to say an Nvidia or ATI hardware video card, it is talking to an abstracted ‘RDP’ video card. Similarly the mouse and keyboard are abstracted to come down the RDP protocol connection. So it still talks the X protocol, so that typical X client programs (eg. xterm) still think they’re talking to an X server, but ‘out the other side’ it’s all RDP traffic. I don’t think it talks RDP over a network connection though (not sure). I think the role of libxup.so is to provide the ‘link’ to it.

So where do you get X11rdp? I couldn’t find many references to it on the net but came across this post that mentions that there is an svn repository containing a modified Xorg 7.1 server that when compiled produces the X11rdp binary. Compiling the old Xorg drivers takes a long long time, but I had success following those notes. So here’s what I did;

cd some_temp_dir_with_plenty_of_space
svn co svn://server1.xrdp.org/srv/svn/repos/main/x11rdp_xorg71
cd x11rdp_xord71
mkdir /opt/X11rdp
sh buildx.sh /opt/X11rdp
# and now wait a long long time

That post on linuxquestions.org mentioned some changes to font directory paths inside the buildx.sh script. I didn’t have to change it at all the first time I tried compiling. I used a Debian Lenny 64 bit host to compile. However, I wanted to put this on a Centos 5.3 32 bit system as well and it wasn’t so good. basically there are a couple of changes to the buildx.sh script. First it compiled libfreetype as a 64 bit binary, so I had to add in the CC=”gcc -m32″ bit below

# freetype
if ! test -f $PCFILEDIR/freetype2.pc
then
  cd freetype-2.1.10
  CC="gcc -m32" ./configure --prefix=$PREFIXDIR
  if ! test $? -eq 0
  then
    echo "error freetype"

And the font path in Centos is a bit different to what the script was expecting, so I had to do the following (adding in the last elif bit)

# make a symbolic link to your local font directory
if ! test -d $X11RDPBASE/lib/X11/fonts
then
  if test -d /usr/share/fonts/X11
  then
    ln -s /usr/share/fonts/X11 $X11RDPBASE/lib/X11/fonts
  elif test -d /usr/X11R6/lib/X11/fonts
  then
    ln -s /usr/X11R6/lib/X11/fonts $X11RDPBASE/lib/X11/fonts
  elif test -d /usr/share/X11/fonts
  then
    ln -s /usr/share/X11/fonts $X11RDPBASE/lib/X11/fonts
  fi
fi

I also compiled for a Debian 32 bit build and had to put ‘CC=”gcc -m32″;export CC’ at the very top of the build script (but that might be because of the virtual machine I was compiling in).

So once you have X11rdp compiled, make sure it’s in your path (perhaps create a symlink from /usr/bin/X11rdp to wherever you have it).

So for me xrdp plus X11rdp works great, but you could never copy and paste between your local environment and your remote environment … until now. As well as the stable releases of xrdp, there is also a CVS repository, and one of the more recent commits (sept 2009) is to add clipboard support (thank you Jay). To get and compile the cvs version you need to check it out.

mkdir workdir
cd workdir
cvs -d:pserver:anonymous@xrdp.cvs.sourceforge.net:/cvsroot/xrdp checkout xrdp
cd xrdp
./bootstrap
./configure
make install

You might have to do some fiddling with various paths to get it all going (most distro’s seem to run xrdp as a non-root user. If you run the cvs version this way it will have trouble writing the pid file when xrdp starts … unless you change the path to the pid file … or do a ./configure –prefix=/usr/local/xrdp … or similar. The best way is to just use strace when launching the binaries to see what’s going on)

You run xrdp then xrdp-sesman (which used to be called just ‘sesman’). The easy way is to run both as root, though you might want to invest the time to get xrdp to run as a non-root user. For example;

su - xrdp -c /usr/local/sbin/xrdp
/usr/local/sbin/xrdp-sesman

The clipboard support just ‘works’ for the most part. I had a few issues with using the Windows XP mstsc client, but rdesktop and the Windows 7 mstsc seem to work fine (UPDATE: It’s only the XP SP2 mstsc that I can’t get to work with the clipboard. The mstsc out of XP SP3 works fine … and in fact works fine if you copy it to a SP2 machine)

(Note: This post was initially written when ESXi 4.0 was available. As of late 2010, ESXi 4.1 has been released, and it does actually include a sky2 driver that may or may not work with various Marvell LAN chipsets. The post is still relevant (especially the comments)  if your particular Marvell chipset does not work with the sky2 driver in ESXI 4.1. Also, the post is relevant if you’re interested in porting other network drivers to ESXi)

Well, after somehow getting my Marvell LAN card working with ESXi 3.5u4 (and u3) I thought I’d have a look at  ESXi 4. Again I somehow got it to go. I’m not too sure how good it works, but it works well enough for me at home. If you can’t be bothered reading about me going on and on and on and on about how to compile it, then just scroll to the bottom of the post. The download for the source includes a precompiled module (NB: As per the post about ESXi 3.5, this is all about getting a 88E8053 chipset Marvell LAN working).

ESX/ESXi 4 is quite different from 3.5. The build chain is similar to 64 bit Redhat/Centos 5.2, so I ended up installing a x86_64 Centos 5.3 inside a vmware fusion machine to do my dev work. I just made sure I installed all the dev stuff.  Then I downloaded the VMware-esx-public-source-4.0-162945.tar.gz from VMware’s open source page. It’s a much bigger file (590MB) than the file for 3.5. When you extract the file you end up with a lot of rpm files plus a vmkdrivers-gpl.tgz file. I did the following to extract it all on my test machine;

cd ~
mkdir vmware-oss
cd vmware-oss
tar xvzf ~/VMware-esx-public-source-4.0-162945.tar.gz
mkdir drivers
cd drivers
tar xvzf ../vmkdrivers-gpl.tgz

One of the rpm files included is a kernel source rpm. I’m not exactly sure what it is relevant to ESX, but I installed it anyway for reference. I found I needed the qt-devel and gtk2-devel packages first;

cd ~
cd vmware-oss
yum install qt-devel
yum install gtk2-devel
rpm -iv kernel-sourcecode-400.2.6.18-128.1.1.0.4.159770.x86_64.rpm

I’m pretty sure you don’t need the kernel source to build the drivers, but I kept it handy for reference anyway.

You can try doing a test build of the drivers now. This will build all the drivers built in to ESX/ESXi.

cd ~/vmware-oss/drivers
./build-vmkdrivers.sh

You’ll probably get a few warnings, but it should complete. If you do a find down the ‘bora’ directory you should see a bunch of .o files corresponding to the kernel modules (look under the ‘bora/build/scons/build’ directory).

OK, my approach was to look at the build-vmkdrivers.sh script and basically look at what was done to compile one network driver (I used the forcedeth driver as a reference) and just make a reduced script for my sky2 driver.  As for the source to base the sky2 driver on, instead of using a driver from 2.4.37 like I did with the 3.5 version of the network driver, this time I ended up using the sky2 source from 2.6.26 (or the debian lenny incantation of it). I did originally use the sky2 driver from the kernel-sourcecode…2.6.18…rpm file, but on closer inspection of the tg3 driver that ESX uses, I noticed it actually comes from a 2.6.24.1 kernel (or thereabouts) … so I thought I may as well use a more modern reference source. I had a few minor hiccups trying to get it to compile, but in the end I just had the following shoved into the top of my sky2.c file;

/* Stuff for ESX compile */
#define upper_32_bits(n) ((u32)(((n) >> 16) >> 16))
#define csum_offset csum
#define bool int
#define PTR_ALIGN(p, a)         ((typeof(p))ALIGN((unsigned long)(p), (a)))

u32 bitreverse(u32 x)
{
        x = (x >> 16) | (x << 16);
        x = (x >> 8 & 0x00ff00ff) | (x << 8 & 0xff00ff00);
        x = (x >> 4 & 0x0f0f0f0f) | (x << 4 & 0xf0f0f0f0);
        x = (x >> 2 & 0x33333333) | (x << 2 & 0xcccccccc);
        x = (x >> 1 & 0x55555555) | (x << 1 & 0xaaaaaaaa);
        return x;
}

The define’s are to remedy compilation errors, and the bitreverse is to satisfy an undefined symbol problem. Note that the undefined symbols errors end up in /var/log/messages on your ESXi 4 box now (unlike 3.5).

So in the end to compile my driver I did a;

./build-sky2.sh

You get a couple of warnings, but if you do a ‘find . -name sky2.o’ you should end up with two sky2.o files. There is a DEBUG and DASHG variable defined at the top of the build script. If you uncomment these it’ll build a lot of debug stuff into the modules.

If you get some errors, its probably because some directories are missing in the build path, so make them first;

mkdir -p bora/build/scons/build/vmkdriver-sky2.o/release/vmkernel64/SUBDIRS/vmkdrivers/src26/drivers/net/sky2
mkdir -p bora/build/scons/build/vmkdriver-sky2.o/release/vmkernel64/SUBDIRS/vmkdrivers/src26/common/

Now, again I used a USB stick with ESXi 4 (build 171294 in my case). To install it, I loopback mounted the VMware ESXi 4 iso file file, extracted the image.tgz file to a temp directory, bunzip2′d the big dd image file, then dd’d it to the whole USB stick (NB: There are some details about how to do this in linux on my other post re ESXi 3.5. See link at the top of the post)

A difference this time is that I didn’t have a simple.map file all pre-prepared to go into the oem.tgz file. I thought the easiest way to get it would be to just boot ESXi and let it fail when it tries to configure a network device, then somehow copy the simple.map file off. So I did this. ESXi 4 merrily boots and eventually you see the dreaded ‘lvmdriver failed’ message. It looks like ESXi is broken at that point, but just type the word ‘unsupported’ and you get a password prompt, and just hit ENTER to get a prompt (You might need to hit alt-f1 first before typing ‘unsupported’)

Because networking is not working, we’ll just copy the simple.map to the Hypervisor1 partition;

cp /etc/vmware/simple.map /vmfs/volumes/Hypervisor1

I just did a ‘sync’ and held in  the power switch ( perhaps type ‘reboot’ if you feel like being more careful). Now get the USB stick to appear as a USB device in your development VM (your centos 5.x environment), and mount partition 5 (or the Hypervisor1 partition) off the USB drive, and you should see an oem.tgz file as well as the simple.map file. You need to make a directory structure up for the new oem.tgz file we’ll be creating;

cd ~
mkdir vmtest
cd vmtest
mkdir -p etc/vmware
mkdir -p usr/lib/vmware/vmkmod

Copy the simple.map off the USB drive into etc/vmware directory in our tree structure. eg.

cp /mnt/simple.map ~/vmtest/etc/vmware

And edit the simple.map so that it includes the PCI ids for your Marvell card. Mine is 11ab:4362 so I added in the bolded line below, but yours could likely be different. If you’re not sure, you could boot ESXi off the USB stick again, do the ‘unsupported’ thing to get a prompt and type lspci -v

1166:0410 0000:0000 storage sata_svw.o
1166:0411 0000:0000 storage sata_svw.o
11ab:4362 0000:0000 network sky2.o
14e4:1600 0000:0000 network tg3.o

Now copy in the sky2.o file that we compiled earlier. The modules are in a different directory compared to 3.5 (NB: the compilation process produces two sky2.o files, so make sure you grab the one shown below)

cd ~/vmware-oss/drivers
cp ./bora/build/scons/build/vmkdriver-sky2.o/release/vmkernel64/sky2.o ~/vmtest/usr/lib/vmware/vmkmod

Now tar it up, and copy it to the USB stick that should be still mounted;

cd ~/vmtest
tar cvzf ~/oem.tgz *
cp ../oem.tgz /mnt

Unmount the USB stick

umount /mnt

Now try booting again. Hopefully you should see a ‘loading sky2′ flash up early in the boot … and it should eventually get to the usual ESX status screen showing the current mgmt IP address. Basically if you don’t see the ‘lvmdriver load failed’ then there’s a good chance it’s working.

And yes here is the sky2-for-esxi4-0.01.tar.gz download. It includes the build script, the modified source, plus directory tree for creating the oem.tgz file including a pre-compiled copy of the module. If you can’t be bothered compiling, you can just extract this file, cd to the vmtest directory and create the oem.tgz file as per the earlier notes.

UPDATE: (2010/02/08) There is also now a driver for the Marvell 88E8001 LAN chipset (see the comments discussion below). This uses the skge driver, not the sky2 driver mentioned above. I don’t own a 88E8001, so thank you to samarium for helping out re the unresolved symbols. Please comment below if you’ve tried the skge driver and it works. I have a new tarball sky2-and-skge-for-esxi4-0.02.tar.gz containing both the sky2 and skge driver

UPDATE: (2010/12/24). xieliwei has a driver that should support the 88E8057 on ESXi 4.1 (as per the latest comments, it seems a bit iffy on 4.0). Anyone else who has this chipset, if you could try the driver and post more information that would be great. The 88E8057 code is here as a local copy ; sky2-1.22-for-esxi4-88e8057-r1-xieliwei.tar.gz or on mediafire; sky2-1.22-for-esxi4-88e8057-r1-xieliwei.tar.gz . Also there’s another version of the driver that will produce copious debug information (only of use if you’re having problems); http://www.mediafire.com/file/m836gce3r3b7ygi/sky2-1.22-for-esxi4-88e8057-r1-oem-debug-xieliwei.tgz

I’ve been setting up an ESXi system for a client recently, and I must admit there is a lot to like about ESXi. It pretty much works like it says in the brochure The client system is a proper Dell server type system and all the hardware is supported on ESXi (3.5u4 in this case). Of course, I’d like to play around with ESXi at home, but I don’t really have any hardware that supports it except my old Thinkpad T42. The better choice would be my core 2 duo tower system but it really needs an Intel gigabit LAN card to be of any use with ESXi, as the onboard Marvell (yukon2) LAN chipset is not supported by ESXi. (NB: The Marvell LAN I have is a 88E8053 on an old Asus P5LD2 motherboard)

However, there are a few projects on the net to port drivers to ESXi since the driver layer is based on some older linux kernels as far as I can tell. I found the open-vdrivers project on sourceforge that has some modules for some Realtek gigabit LAN cards for using with ESXi 3.5. There is also some info on the open-vdrivers site re setting up the build environment to compile the  drivers  for ESXi 3.5. Apparently you need an old gcc compiler (3.2 or 3.3 I think), and the recommendation was to use Centos 3 as a platform. It appeared as though they had taken the realtek driver source from a 2.6.18 kernel and modified it to get it to go in the VMware build environment.

So I thought I’d attempt to port the linux sky2 driver (what the Marvell Yukon2 LAN chipsets use) compiled and running on ESXi 3.5.

How hard could it be? (I must admit I’m a hackerish C programmer at the best of times and didn’t have high hopes).

Amazingly I got it to work. I honestly don’t know how well it works, but I’ve performed backups over the network with it and it hasn’t died.

So I installed Centos 3.9 i386 in a virtual machine, ticking the boxes to make sure I had the dev environment stuff installed (I ended up needing all three install CDs). I also needed subversion to check out the open-vdrivers project and I must admit trying to find subversion for Centos 3 was painful (I think I ended up finding a Subversion 1.2 rpm somewhere that worked). It’s probably easier to check out open-vdrivers on a more recent distro and then just copy the directory tree into your Centos 3 VM.

The notes on setting up the build enviornment for open-vdrivers is here at the moment. I’ll repeat a fair chunk of this below. You need the open sourced files from Vmware to get started. I was using 3.5u3 initially (since that was the only install iso I had), but later repeated the process with 3.5u4. All the notes below will just refer to 3.5u4. And yes, yes, I know 4i is out, but I haven’t even attempted to look at 4i yet.

So go to the VMware open source page and download the 3.5u4 OSS source code. The file I ended up with was VMware-eesx-public-source-3.5.0-153875.tar.gz. Extract that file in your Centos3 machine and you should end up with a bunch of other tar.gz files;

VMware-eesx-gpl-public-source-3.5.0-153875.tar.gz
VMware-esx-server-public-source-3.5.0-153875.tar.gz
VMware-esx-drivers-public-source-3.5.0-153875.tar.gz

Extract all those in a temporary directory. You’ll end up with a few more tgz files which aren’t that important as well as two directories;

VMware-esx-drivers-public-source-3.5.0-153875
VMware-esx-public-source-3.5.0-153875

Check out the open-vdrivers project inside the same temp dir (or check it out on another machine and then transfer the ‘open-vdrivers’ directory to your centos3 machine.

svn co https://open-vdrivers.svn.sourceforge.net/svnroot/open-vdrivers/trunk open-vdrivers

First you have to add a symlink into the VMware source distribution. In my case;

cd VMware-esx-drivers-public-source-3.5.0-153875/src/include
ln -s ../../../VMware-esx-public-source-3.5.0-153875/include/lkhdrs/asm-i386

Now cd into the VMware-esx-drivers-public-source-3.5.0-153875 directory inside your temp directory and execute the build script for VMware’s own open sourced drivers;

cd ~/tmp/VMware-esx-drivers-public-source-3.5.0-153875
./build-vmkdrivers.sh

You’ll get a fair few warnings and mine always ends in an errors about the usb-storage driver (see below). I don’t really care about that, and I don’t think it really matters for what you’re about to do.

**** Creating library drivers-usb-storage-obj.a
ar: build/release/drivers/usb/storage/debug.o: No such file or directory
ld: cannot open build/release/drivers/usb/storage/bin/drivers-usb-storage-obj.a: No such file or directory

Now we need to do a test compile of the open-vdrivers code. First you need to set up another symlink;

cd ~/tmp/open-vdrivers/src
ln -s ../../VMware-esx-drivers-public-source-3.5.0-153875/src/include

And then try compiling one of the realtek drivers to check that it succeeds;

./build-r8169.sh
**   Compiling        drivers/net/r8169/r8169.c
**   Compiling        drivers/net/r8169/linux_module_heap.c
**** Creating library drivers-net-r8169-obj.a

The actual module is in  build/release/drivers/net/r8169/bin/

ls -l  build/release/drivers/net/r8169/bin/r8169.o
-rw-r--r--    1 root     root       979049 Aug 14 23:01 build/release/drivers/net/r8169/bin/r8169.o

OK, so far so good. Now, if you just want to try compiling my sky2 build, grab my sky2-for-esxi35u4-0.01.tar.gz and extract it into the open-vdrivers directory, then compile it;

./build-sky2.sh

And then you should end up with the driver in build/release/drivers/net/sky2/bin;

-rw-r--r--    1 root     root       975909 Aug 14 23:15 sky2.o

Yeah, I’m not so sure why its that big either. To get it to work on ESXi, I find it much easier if ESXi is on a USB stick. There’s a few howtos on the net for how to do this. Most seem to be Windows-centric. Here’s my brief linux take on it;

cd /tmp
mkdir vmiso
mount -o loop VMware-VMvisor-InstallerCD-3.5.0_Update_4-153875.i386.iso /tmp/vmiso
mkdir /tmp/install
cd /tmp/install
tar xvzf /tmp/vmiso/install.tgz
cd usr/lib/vmware/installer
# Insert a USB stick of at least 1GB in size. Make sure it doesn't mount. If your linux distro auto
# mounts it then unmount it. Run dmesg to find out what its device name is. In my case it was sdd,
# so I put of=/dev/sdd below. This will vary depending on your machine. MAKE SURE YOU GET IT
# RIGHT, otherwise you'll trash one of your harddisks)
bunzip2 -c VMware-VMvisor-big-3.5.0_Update_4-153875.i386.dd.bz2 | dd of=/dev/sdd bs=32k

That bunzip2/dd thing will take a while. Wait until your USB stick activity LED stops flashing and then probably pull it out and stick it back in again. If you do a dmesg again you might see some partiion info;

[560078.250013]  sdd: sdd1 < sdd5 sdd6 sdd7 sdd8 > sdd4

So that gets you a default install. Now how do we insert the sky2 driver into it? You need to firstly get the sky2.o driver onto the USB stick and then you need to make sure the PCI id information is updated so that it loads the correct driver at boot time. If you’re currently running linux on the machine you want to use for ESXi, you can run lspci to work out the PCI id for your card. On mine;

# lspci -nn

...
02:00.0 Ethernet controller [0200]: Marvell Technology Group Ltd. 88E8053 PCI-E Gigabit Ethernet Controller [11ab:4362] (rev 19)
...

The 11ab:4362 bit is my PCI id. You need to write this down.

There are few possibilities for which tar.gz file to change on the USB stick, but I used the oem.tgz method. In my case I had already downloaded the Unified oem.tgz from acronymlabs . This oem.tgz is actually for ESXi 3.5u3, but it works well enough for me on u4.  Extract that oem.tgz file into a temp directory on your linux box,

cd /tmp
mkdir oemstuff
cd oemstuff
tar xvzf ~/CommunityUnifiedDriverPack_v1.1.0_U3-123629.oem.tgz

then edit the etc/vmware/simple.map file and add an entry in for your PCI id. In my case, I added the bolded entry below;

1166:024b 0000:0000 storage sata_svw
1166:0411 0000:0000 storage ide
11ab:4362 0000:0000 network sky2
13c1:1003 0000:0000 storage 3w_9690

Now copy the build/release/drivers/net/sky2/bin/sky2.o file from your build directory into the mod directory

And create a new tar.gz file and make sure its called oem.tgz

cd /tmp/oemstuff
tar cvzf /tmp/oem.tgz *

Insert the USB stick and mount partition 5 (or the one with the label Hypervisor1)

mkdir /tmp/usb
mount LABEL=Hypervisor1 /tmp/mnt

You’ll see that there already should be a tiny oem.tgz file in that partition. You can back it up if you like, but there is pretty much nothing in it. So just copy your new oem.tgz file to the USB stick

cp /tmp/oem.tgz /tmp/mnt

And unmount it

umount /tmp/mnt

Now try booting off the USB stick (you might need to change your BIOS settings for USB booting, or I ended up pressing F8 to get the boot select menu most of the time). I disconnected all my linux hard disks before rebooting as a precaution. You can boot off the USB stick with no physical hard drives in your system, and on this first boot we just want to confirm that the sky2 card is recognised.

So you should see the ESXi boot up messages and some progress info about loading drivers. If you see ‘lvmdriver failed to load’, then that usually means your network card driver didn’t load. Once all the boot up stuff has finished you should get the status screen with an IP address onscreen (probably 0.0.0.0 initially). Now you can press Alt-F1 and type the word ‘unsupported’  (you won’t see any of these letters as you type it). Then you’ll get a root password prompt. just hit ENTER as you probably haven’t assigned a root password yet. Now you should have # root prompt. Have a look at the /var/log/config.log file;

less /var/log/config.log

Scroll through the file, and find where its loading the sky2 driver.Hopefully you should see something like;

Using /mod/sky2.o
Module load of sky2 succeeded

If you don’t, or you see messages about unresolved symbols then it hasn’t worked. If it has worked, then you can ‘exit’ and press Alt-F2 to go back to the main status screen. Press F2 to configure ESXi. Assuming the network card is working, you should be able to configure an IP address for the mgmt interface as well as a hostname. Go to a machine running Windows, overcome your depression, and point a browser at https://<the IP you assigned to your ESXi box> and you’ll get a cert warning and then a default page that allows you to download the Infrastructure client from ESXi itself. Install that and configure your ESXi box.

For reference, I started out using the sky2 driver from a 2.6.18 kernel (since the realtek drivers seem to use this version as a base), but I wasn’t making a lot of progress, so then I read that ESXi 3.5 is closer to a 2.4.21 kernel, so I used the most recent 2.4 kernel (2.4.37) and grabbed the sky2 driver from it. That turned out to be a lot quicker to get to the point where it actually compiled successfully (NB: the final ar in the build script will always fail unless the ‘bin’ directory exists for the output file). Once I had it successfully compiled, it still failed to work in ESXi. Looking at the /var/log/config.log I was getting a few unresolved symbols alerts when it tried to load the sky2 module. I noted down the names of these symbols, then went back to my centos3 build system and searched a vanilla 2.4.37 kernel tree looking for these missing symbols. Fortunately they tended to be simple functions that had no further dependencies, so I basically just copied the code from the kernel directly into the driver and recompiled, put it all on the USB stick again and rebooted until the module loaded cleanly … and then amazingly it worked. I also tried hacking at the 3c59x driver in order to get a 3com 3c905 boomerang card to work (which seems to be the one type of 3c905 that is not supported by ESXi). Again I got it to compile and load, but honestly I don’t think it works properly. I had it working sort of OK as a secondary interface, but if ESXi decided to use it as vmnic0 (ie. the mgmt interface nic) then ESXi would crash on boot.

And finally a big disclaimer. I am certainly not a kernel driver programmer by any stretch. I somehow got this driver to compile and amazingly it works fine on my home network in the limited testing I’ve done. I still think for production use, go buy some intel pro 1000 LAN cards or get a motherboard with an intel gigabit lan integrated.

UPDATE: 2010/03/27. On my ‘Using a Marvell LAN card with Vmware ESXi 4′ post there’s been some discussion about using a different type of Marvell card; a 88E8001 which uses the linux skge driver. I’ve finally had some feedback that my latest attempt at an skge driver might be working. Thats under ESXi4. So I’ve had an attempt at doing much the same hackery of the linux 2.4.x skge drive for ESXi3.5 . So if you have a 88E8001 and want to try it with ESXi 3.5, grab skge-for-esxi3.5u4-0.04.tar.gz and post some feedback as to whether it works or not. You’ll need to edit the simple.map file and add an entry like;

11ab:4320 0000:0000 network skge.o

If the module does not work, please post any ‘unresolved symbol’ info you find in the logs. As per the other post, I don’t own an 88E8001 based device so cannot test it myself. This v0.04 release adds some suggestions by ‘Sheepless’ (see comments), but I am dubious that it will actually work. As per Sheepless’ comments, this skge driver might be a bit harder to do for ESXi 3.5. But, if you’re able to try it, please do, and post some feedback.

I’ve had my Macbook almost one year now. When I bought it (an apple refurbished model), I noticed that a lot of keys didn’t register very well, and I ended up replacing the macbook keyboard. That second hand replacement keyboard has served me well for the past 10 months or so, but recently it’s started to fall apart. First it started as a small chip breaking off in the lower right corner of the front face, but a week later, there was a long crack near the chip where I knew the whole thing would break off eventually.

So yes the keyboard still worked, but it looked pretty awful. I’m often at client sites with the laptop and it is slightly embarrassing to turn up with a broken looking laptop. So off to ebay we go, and I bought yet another replacement keyboard. Three keyboards in 12 months is pretty sad. I quite like OSX as a platform, but apple hardware quality is definitely debatable.

Now, the original keyboard on my macbook is what is often called a V3 keyboard (at least that’s what a lot of ebay sellers call it). It has the mute/softer/louder buttons over F10, F11 and F12. A V2 keyboard has the mute/softer/louder over the F3, F4 and F5 keys. There’s a few other differences as well. My 2nd keyboard (the one now with the crack in it) was a V2 keyboard. It was in good condition when I bought it. I probably prefer the layout of the V2 keyboard over the V3, but this time I ended up buying another V3 keyboard on ebay. It was a factory second with some scuff marks on it that you can only see if you angle the keyboard in the light a certain way. My thoughts are that its effectively brand new, so hopefully it will last more than 10 months.

Warning; The discussion below relates to my attempts to use rsync to back up a directory on a MAC to a directory on a remote linux host. I never worked out how to get it to back up 100% of all the unusual metadata that MAC filesystems include, but it works well enough for my purposes.

I’ve been using rsync for quite some time to backup directories between linux and other types of Unix systems. It works great. It’s often as simple as;

rsync -av -e ssh source_dir user@remotehost:destdir

Little did I know, rsync to/from a Mac to a non-Mac unix is actually quite complicated. Unlike most other Unix filesystems, the HFS+ filesystem that Macs use includes a bunch of extra ‘metadata’ such as resource forks, finder-flags, finder-locks, a special type of creation-date, bsd-flags etc. Obviously, if you’re copying a Mac directory elsewhere for backup purposes you’d like to be able to restore all your files plus this ‘extra metadata’.

Apparently the oldish version of rsync (v2.6.9 on my OSX 10.5.7) has the Apple specific -E special option to allow it to capture this extra ‘metadata’, but it only really works if you’re copying to another directory on the same server, or to another Mac running the same modded Apple rsync. Apparently, the more recent versions of rsync (latest is 3.0.6) include features that allow them to save/restore most of the OSX specific ‘metadata’.

I ended up installing the latest version of rsync on my macbook using macports. Interestingly, it is v3.0.5 plus some additional patches for OSX (fileflags and crtimes patches). It installs under /opt/local/bin. I also installed backup-bouncer which can create an OSX volume containing a set of test files, each with some of the more unusual features of OSX. You then use your directory copying program to copy the test files elsewhere and then use backup-bouncer to check if all the OSX metadata were copied across.

Using the macports rsync, it is really good at rsyncing between two local directories or to a directory on another Mac. For example, if I do the following, backup-bouncer’s verify tells me my ‘testdir’ passes all tests;

sudo /opt/local/bin/rsync -av --crtimes --hard-links --acls --xattrs --fileflags /Volumes/Src testdir

However, I can’t do that when transferring to a non-mac. rsync generally requires that the version of rsync at the destination end also understands your command line arguments. I attempted to compile rsync on my Debian Lenny server using the same build technique the macports version uses (ie. with the crtimes and fileflags patches). This does not work, since those two extra patches require a bunch of OSX specific header files. So that left me with compiling the normal rsync 3.0.5 from http://www.samba.org/rsync/download.html. That compiled fine on my Debian box, and I ended up copying it over the /usr/bin/rsync that I already had on the server.

So I couldn’t use the crtimes or fileflags patches. Backup-bouncer doesn’t class those things as ‘critical’, so I thought from a backup point of view, I’d leave them out. So I tried this to copy data to the linux box ‘host’

/opt/local/bin/rsync -av --hard-links --acls --xattrs -e ssh /Volumes/Src user@host:/backup

That also did not work. The /backup filesystem on the linux host was ext3, but you need to use additional mount options if you want to use extended attributes or ACLs, so I remounted my backup partition;

umount /backup
mount -t ext3 -o user_xattr,acl /dev/hdc1 /backup

Now, it works a bit better, but a lot of stuff still fails. The basic problem is that there are still a bunch of OSX specific metadata bits that rsync on the Mac can read, but the linux end does not know how to store them. For this, rsync has the –fake-super option. It basically encodes the extra stuff into the extended attributes on the remote system.

I finally ended up with the following that MOSTLY works. It still fails on some of the non-critical backup-bouncer tests. It also cannot handle large amounts of metadata

• Back up a directory from the Mac using;

  /opt/local/bin/rsync -aHv --acls --xattrs -e ssh  \
  --rsync-path="rsync --fake-super"  source_dir user@linuxhost:/backup/latest

• Restore a directory using.

  /opt/local/bin/rsync -aHv --acls --xattrs -e ssh \
  --rsync-path="rsync --fake-super"   user@linuxhost:/backup/latest restore_dir

One thing I have read is that the size that ext3 allows for extended attributes is quite small, and if you have a lot of metadata for a file, then this small space can fill up. I’ve noticed this with backups of photos. The resource fork info for each photo is quite substantial and I get alerts like this when I run rsync;

rsync: rsync_xal_set: lsetxattr("Pictures/iPhoto Library/Originals/2008/8:11:2008/IMG_0023.JPG",
"user.com.apple.ResourceFork") failed: No space left on device (28)

On OSX, you can examine attributes with the OSX xattr command;

$ cd 'Pictures/iPhoto Library/Originals/2008/8:11:2008/'

$ xattr IMG_0023.JPG
com.apple.FinderInfo
com.apple.ResourceFork
com.apple.metadata:kMDItemSupportFileType

$ xattr -pl com.apple.ResourceFork IMG_0023.JPG

When you enter that last command you end up with a hex dump of the resourcefork thing. My output ends with B0B0 as the last line … which is about 45Kbytes. I think ext3 has a limit of 3.9K for all extended attributes, hence the error. I found this post that suggested using XFS as it can handle much larger extended attribute data (maybe 64K).

I tried XFS, but ultimately gave up on it, since I always had bad delete performance. Now, I’ve settled on reiserfs. It can also handled a large extended attribute set (NB: You have to specify acl,user_xattr with reiserfs, just like the examples for ext3). I still get a few errors with my backups re issues with backing up metadata, but it works well enough for my purposes.

Google Web Toolkit (GWT)  has a great development environment using the built in jetty server, but the default setup is aimed at developing servlet style server-side programs. If you want your GWT client to simple do some GETs or POSTs to a CGI script somewhere, my initial thoughts were to use a different web server on a different port, but then you hit the SOP (single origin policy) problem. Basically any server communication needs to be to the same web server (ie. the jetty one).

The simple answer for me was to tell the jetty server to serve CGI scripts. To do this, edit your war/WEB-INF/web.xml file so that the following lines are added in;

<servlet>
    <servlet-name>perl</servlet-name>
    <servlet-class>org.mortbay.servlet.CGI</servlet-class>
</servlet>

 <servlet-mapping>
    <servlet-name>perl</servlet-name>
    <url-pattern>/cgi-bin/*</url-pattern>
</servlet-mapping>

Now create a cgi-bin directory underneath your war directory and create some perl scripts in there. You’ll need to make them executable, but that’s pretty  much it (I did this under OSX). The only other key problem I had is that when you create a new GWT project in eclipse you have to untick the option for Google App Engine, otherwise the jetty changes don’t seem to work.

Well, after my excitement over KVM, I somehow found myself looking at vmware’s desktop products again (Workstation, Server, Player and Fusion). I use Fusion on my Macbook and it’s always worked great. I’ve never even had to consider trying to tweak anything to improve performance with Fusion. It’s always been great. Constrast that with running Server or Player on my core2duo Linux box. As I said in an earlier post, I’ve never been able to get them to work very well, especially when you attempted to run more than one virtual machine. This machine has 2GB of ram and the performance degradation that occurred when you put two busy VM’s running on it was pretty bad. I’m sure I’ve looked at the MemTrimRate and various other tweakable parameters and never really found anything that helped.

So recently I got sidetracked looking at this. It requires using one of VMware’s desktop products, so I started having a look at Vmware player on linux (v2.5.2 is the latest). It didn’t take me long to get it going, and I must admit it works pretty damn well even though I never bothered to set sound up (there are some good hints further on in that thread about key repeat issues). But of course, I wanted to try running another Vmplayer session at the same time, at which time I was reminded how well virtualization turns to crap on this decent spec machine when you run two or more VMs on vmware player.

So I started googling … and there’s a lot of info out there, as Vmware Player/Workstation/Server all exhibit much the same behaviour under linux. Much of the good info relates to Vmware Server. The most useful post was this one, whish talks about a few things to do. And this one and this one was useful too. I ended up using a combo of some of the settings they mentioned. The main ones that I found particularly useful were;

In /etc/vmware/config;

tmpDirectory = “/tmp/vmware”
mainMem.useNamedFile = “FALSE”

And then mkdir /tmp/vmware and mount /tmp/vmware with the following in my /etc/fstab

tmpfs /tmp/vmware tmpfs defaults,size=3G 0 0

And also to add the following /etc/sysctl.conf settings

vm.swappiness = 0
vm.overcommit_memory = 1
vm.dirty_background_ratio = 5
vm.dirty_ratio = 100    (actually, later I ended up changing this to 80)

Those posts obviously mention a few other settings. There’s quite a few that relate to Vmware’s tricks re memory sizing up and down rather than say a 1GB RAM VM actually consuming 1GB of RAM. I was actually keen to keep some of this auto-memory resizing stuff at the expense of performance.

Well, these changes do make a marked difference for me. I can even convince it to run two VM’s each with 1GB of RAM on my 2GB of ram server … and it a) lets me do this and b) doesn’t go into swappiness madness like it used to do.

There’s actually an interesting comment in that last link I mentioned from someone who works on the memory subsystem (for vmware I assume). There’s an interesting quote which might explain why VMs seem ‘much smoother on OSX’ than they do in Linux;

The big-picture answer is that the Linux virtual memory subsystem is simply not tuned for running VMs

I have to admit Linux’s swapping behaviour does seem somewhat more painful than other OS’s at times.

Alas, being a glutton for punishment, I wanted to push things, and ran a 2nd VM (Windows 7 RC 64 bit) with 1GB of configured RAM and a 3rd VM (Ubuntu 9.04 64 bit) with 512MB of RAM. So that meant 3 guests with a configured total guest RAM of 2.5GB on my core2duo with 2GB of RAM. Yes, I didn’t think the outcome would be pretty, but I thought Vmware might be able to handle it.

Now, I only lightly loaded these VM’s and it wasn’t too long before my box went into paging hell. Some interesting commands to run are;

vmstat 5

And watch the si and so columns, and also;

watch grep -A 1 dirty /proc/vmstat

I picked up the latter command from this interesting performance troubleshooting page. It better explains how the output relates to the dirty_ratio and dirty_background_ratio.

Even though the 3 VM’s were operational, the major problem was that occasionally one of the VMs would disappear. A closer look at dmesg revealed that the OOM killer had stepped in. I’d never heard of the OOM killer before. Basically the kernel monitors whats left of ‘low memory’ in 32 bit linux. ‘Low memory’ is memory in the first 900MB or so of your system. It seems to have some special purpose, and it’s important that you never have NO ‘Low memory’ left. So the kernel now will go out and kill the process hogging the most ‘low memory’ if it detects that it might run out. You can apparently turn it off, but the bigger problem is that you can run out in the first place regardless of how much RAM you have. A key workaround is to move to 64 bit linux. My server had run Debian 32 bit (first Etch and later Lenny) for quite some time, and only recently had I been looking at Ubuntu 9.04 64 bit. I was quite enjoying Ubuntu for a while, but bit by bit a few things started to annoy me about it, so I switched back to Lenny 32bit to do all this VMware testing.

So I haven’t really solved this lowmem problem. I could just switch to 64 bit linux to see how it goes. In the meantime, I’m actually going to get some more RAM for this box. I’m interested to see how it performs when there is more RAM available than the total required by the guests. In theory, I might still get lowmem problems but I’ll see how it goes.

My core2duo linux box here has been running a software RAID1 swap and root for some time now. Basically /dev/md1 is the mirrored swap using the first gig of two drives, and the root partition is /dev/md2 using partition 2 on the same drives. It all works quite well.

Recently I was doing some experiments with virtual machines and in an attempt to improve performance I thought I’d change the RAID1 swap to a RAID0 swap. I thought this should be relatively simple. I had another partition elsewhere that I added in as additional swap first, then removed the /dev/md1 swap from the system;

swapoff /dev/md1

And then tried to shut down and delete the /dev/md1 array. This did not work

mdadm --stop /dev/md1
Device or resource busy.

Hmmm, I’ve removed md devices before and I thought this should really work, since the device is no longer used as swap. So off to the google we go. Of course lots of people have similar problems … and it was interesting that a lot seemed to be related to swap. Per a lot of forum discussions, there are questions and not many answers.

However, I found this SUSE forum post and it had the answer that worked for me. Basically I had to reboot, edit the grub boot config and added ‘noresume’ onto the end of the kernel line. Once it was booted up, I could now stop /dev/md1 and remove it. Something to do with software suspend locking the swap device. Crazy. Very un-unixy.

I recently took on some work that involved creating a web frontend to some configuration data on a server. The configuration information was in an XML file, and essentially consisted of a list of server directories and various pieces of information related to each directory.

The goal was to create some web frontend method of editing this XML file. Given that the configuration could list 100′s of directories, the solution I thought needed some tabular way of editing the data. The obvious idea I thought would be to use some AJAX solution. That would give a smooth user experience, and would generally ‘look good’.

The key problems with AJAX are the handling of browser quirks and the complexity of handling asynchronous server requests. These kinds of issues have been mostly solved by the various AJAX Javascript libraries that exist, so I started investigating what was available. There are quite a few, but I focussed on jQuery and Dojo . Each has a plethora of ‘plugins’, some of which can create a ‘Editable Grid’. Essentially these present a table on screen and offer various means of editing the cells in the grid and sending the updates to a server. They’re generally quite configurable. I looked at a few of these ‘Editable grids’, but ended up focussing on a jQuery  one called jqGrid

jqGrid comes with some doco and quite a few examples. It didn’t take too long to hack together a simple setup that allowed me to edit cells and have the updates go to/from a server. However, the type of grid I really wanted to do was reasonably complex and really needed editable cells containing a composite of text boxes, dropdowns and other stuff.

I could see that my changes might be possible … but at the expense of modifying the jqGrid plugin itself to meet my needs. This approach never appeals to me. Both jQuery and jqGrid are still under some development, so my thoughts are always “What happens in a ‘years’ time when this thing needs bugs fixed or a modification made?”. Do you update jQuery and/or jqGrid and then consequently do you need to work out how to patch the more recent jqGrid to keep the thing working? I kept thinking there was probably a better solution (keep in mind that jqGrid and jQuery are rather good, but they didn’t suit my requirements here).

I started looking at Google Web Toolkit (GWT). GWT is unusual in that it is a a sort of a framework for writing AJAX style apps but in Java, not Javascript. No, it’s not java applets all over again, there are no plugins involved on the browser side. Basically GWT effectively compiles your source code written in java into javascript. So you develop your app in Java, and the browser still runs javascript natively to execute your client side app.

But it offers a lot more. To me it allows you to develop a client/server type setup entirely in java as though you were writing a regular local client app to run natively on say Windows (there are some limitations of course). There’s a GUI framework within it that allows you place widgets on screen and handle user interaction too. I guess it abstracts you away from ‘the browser’ and makes you think more in terms of purely client and server.

One of the cooler parts of it is that you can develop using an Eclipse plugin. So you write your code in Eclipse and then it has this thing called Hosted mode where basically you can debug the entire client and server using a built in servlet server (Jetty) and a browser. I assumed the debug cycle would be ; write some code, compile to java script, check the result in a browser …. and repeat. However, in this Hosted Mode it can (most of the time) pick up changes to your code dynamically, so the process becomes; write some code, click refresh in the hosted mode browser … and repeat.

If the server side of your code can run on a servlet server, you can develop it all as part of your project in eclipse and check it as you go. And GWT has its own remote procedure call stuff, so you can mostly abstract away the whole async http request stuff.

And lastly, one thing I like about most of Googles ‘offerings’ is that they bother to write decent doco and intro’s for there stuff. And I think they must get someone to proof read or make it sure it meets some standard. The GWT site has a decent Getting Started Tutorial and other documentation. And hey, you can even tell they updated the tutorial so that it relates to the current version of GWT (1.6.4 as I write this)

So I was excited by GWT, but I had a few problems;

• I don’t know Java.
• I don’t know GWT
• I don’t know Eclipse

So how could I not know Java? As a nerdy type approaching the big four oh, I spent my youth programming in procedural languages and ended up spending a lot of time writing in various assembly languages (6502, Z80, 68000) and then C. I never really caught onto the whole ‘Object Oriented Programming’ thing. I have a few books on Java and have tried to learn a few times in the past, but always lost interest. I am fine with the syntax of the language, but the whole OOP methodology generally baffles me. I am always left with the question of “Why do it this way?” … which most java books seem to avoid the answer to. So by using GWT I’ve kind of forced myself to relearn Java (often a good project is all it takes to learn a new language). My thoughts now are “It’s not too awful”. As a language I kind of think it lets you write really sad looking code, and yet your app doesn’t crash. I guess that helps answer my ‘Why?’ question. I’m still pretty vague on all the various OOP principles though.

GWT is effectively the base of Java plus its own frameworks. There’s a lot of them. It takes some time to get around it all, and to see how everything works together. In the case of my app, the GUI side of things is relatively straightforward but there’s also a lot of XML involved and for whatever reasons I decided to send stuff to and from the server in XML … which meant trying to understand how GWT works with XML.

Previously, my preconception of Eclipse was that it was a fancy text editor written in Java designed to eat all your RAM and CPU resources. Of course, now that I’ve installed it and used it I can see that it is rather handy, especially all the predictive/hinting  stuff it does and the sheer power of its plugin architecture. The big downside is that the user interface is incredibly ‘busy’. The main IDE I’ve been using before this was Xcode … and its very uncluttered in comparison. I can kind of navigate and do what I want in it now, but initially I found it quite hard. It has a lot of ‘metaphors’ I just wasn’t used to. I even had trouble initially just trying to import some of the GWT sample projects. I still think it’s a resource hog (so it helps to have a decent spec machine), but I must admit that it probably does improve productivity. As a side note, I’ve been running all this GWT stuff on my Macbook (running Leopard). It was pretty easy to set up. However, I also tried to get it running on my Ubuntu Jaunty (9.04) 64 bit linux server. I ended up giving up. Essentially GWT is 32bit as far as I could tell, and even though I think it’s possible to get it running on 64bit linux, it looked very very painful.

So, my project is coming along nicely now. I’ve had to write quite a lot of code though. Probably a lot more than if I’d used a javascript library, but I think I have a ‘better solution’ in the long run. This whole exercise has reminded me of the difficulty of learning something new, and the whole ‘steep learning curve’ issue. I had to learn quite a few new (to me) technologies and though initially you often think you’re not making much progress, eventually there is a bit of a ‘hump’ you get over, and its all good thereafter.

KVM in this case is the linux  ‘Kernel-based Virtual Machine’, not Keyboard/Video/Mouse. I’ve always been puzzled why you would name the new whiz bang virtualization technology for linux the same as a rather mundane piece of equipment found in most data centres.

So I’ve been a big virtual machine user for many years. Mainly using the desktop products like Vmware Workstation (and Fusion on the Mac) and I’ve tried Virtualbox a few times. I tried VMware Server a few times but was never happy with its performance on linux. But, Vmware ESX is always quite impressive. Being so low level it obviously has quite a few extra ‘smarts’ over the desktop and ‘on top of a regular OS’ type VM products. It is incredibly popular in data centers, and has the key benefit of ‘being around for years’, and has consequently been refined and refined.

So that brings me to KVM. In theory it should be the linux world alternative to ESX. Sure Xen kind of does the same thing, but KVM is very integrated into the kernel supposedly, whereas Xen has its own microkernel or somesuch. Of course, when KVM first came out, I tried it and it didn’t seem to work very well, so I forgot about it until now. Lately, I’ve been looking at other distros and have been playing quite a bit with Ubuntu 9.04. Usually I would hate something like Ubuntu due to the volume of updates you typically get, but I was interested in ext4 performance and I had also read that KVM was reasonably integrated into it, so it seemed like an easier way to try those things out.

So I installed Ubuntu 9.04 (the 32 bit version initially) on my 2Gb ram core2duo on top of ext4 and read this KVM post at Adoleo to get some hints on setting it up. I must admit there is a lot of ‘not quite’ consistent information on the net about setting up KVM … and I can understand people who have tried setting it up, got stuck, googled for answer … and then given up. The Adoleo post is quite good as it’s quite recent. There were a couple of extra apt-get’s I needed to do to get everything going (you need the qemu package for example).

Now, with all the bits installed, I of course wanted to install a guest. This is where I think a lot of info on the net becomes confusing. It’s very much a case of heaps and heaps of ‘abstractions’. Here’s how I think everything fits together;

• kvm itself is a low level kernel module that does the virtualization stuff using the Intel VT extension or the AMD SVM extension
• qemu is a software  i386 (and others) CPU emulation system that has been around for years and has a lot of tools necessary for doing virtualisation stuff. In the past qemu has been slow.
• qemu now can leverage itself against kvm to do ‘in hardware’ virtualisation. So I think if you run the kvm command (which seems ot take all the same args as the qemu command) on a VT or SVM capable CPU and you just happen to be emulating a intel type PC architecture system then it’ll go and use the dandy fast hardware virtualization stuff.
• Launching these virtual machines using the kvm command at the command line means you end up with a long list of args. Someone must have thought ‘Lets come up with some clever XML abstracted network thingee instead’ and hence you have this libvirt API which seems to be an API for managing virtual machines both locally and across a network. All the args to the kvm command are now in the form of some XML file.
• Someone thought we need some GUI management tool just like vmware etc have and you end up with virt-manager which seems to be used in the majority of examples on the net. virt-manager uses the libvirt API.

Well, all the above is how I think it works … today

So based on that Adoleo info for the basic low level setup, I think I started up virt-manager (you might have to apt-get it too) and you get a window that says;

localhost (System)

localhost (User)

Of course I could not find a howto anywhere that mentioned why there are two entries and what each one is for. What I think they mean is that localhost (System) is for all the virtual machines running as root level , and the localhost (User) is for all the machines running as your username (???). Anyway I clicked the User one and ‘connected’ to it OK, then right clicked to create a new VM. I think I tried a ubuntu 9.04 VM first. That sort of worked OK. You get a dandy graphical console view and even though the mouse was sometimes erratic (there is a hint somewhere to configure the virtual mouse as a usb tablet to get around this) I got the VM installed OK.

Then of course I could not get networking to go. I’ve set up bridging using brctl before so I had an idea in my head how I wanted things to work. I changed my /etc/network/interfaces to bridge my eth0 into a br0 bridge (as per that Adoleo page), and my host side networking worked fine. Guest networking was a no go. it eventually clicked that even though my user had write permission to /dev/net/tun that it probably was not creating the tap devices in the first place etc. I googled for a while, gave up, then thought “I’ll just make the VM’s part of that localhost (system) thing which should run as root and get rid of my network probs.

This meant working out where the libvirt XML config files live for the VMs. They live in ~/.libvirt/qemu. Just mv them from there into /etc/libvirt/qemu, and restart the libvirt daemon (I seem to have to do this restart a lot);

/etc/init.d/libvirt-bin stop

/etc/init.d/libvirt-bin start

Probably shutdown all your VMs too and stop virt-manager and start it again and you should now see all your VMs under ‘localhost (System)’.

Now my networking worked. I had something like this in my XML file for the ubuntu VM;

<interface type=’bridge’>
<mac address=’54:52:00:12:34:56′/>
<source bridge=’br0′/>
<model type=’virtio’/>
</interface>

and also I had the following somewhere for the mouse;

<input type=’tablet’ bus=’usb’/>

So now I tried a Windows XP VM. It also worked. I had some fun with the networking. Initially I used a line like the ones above, which specify that virtio thing. Virtio is some special driver interface that KVM virtual machines can use to get much faster disk and network performance. It requires a special driver in the guest OS. I think in the case of ubuntu 9.04 this is already integrated. With my XP VM I would just get a ‘Ethernet Controller’ device detected … and for the life of me I could not track down where the Windows XP virtio drivers now live on the net. All the links to them on the net seem either broken or really old. So I kind of gave up, and my XP VM now just has a network section like so (without the virtio ref);

<interface type=’bridge’>
<mac address=’54:52:00:65:43:21′/>
<source bridge=’br0′/>
</interface>

In the XP device manager I now just see a Realtek 8139C network device. It is autodetected and configured and networking just works.

So then I tried a Windows 7 RC VM. The only ISO I have is for the 64 bit version. Of course, one of the limitations of KVM is that you cannot run a 64 bit guest on top of a 32 bit OS (I think some of the vmware products can do this). Anyway, my host OS install was relatively new (and destroyable) so I downloaded the 64 bit iso of Ubuntu 9.04 desktop and got that up and running. And set everything up AGAIN.  Now I tried the Windows 7 install and suffice to say it does not work. It starts, it goes a bit of the way through the install and then dies. So far I’ve googled but haven’t found a solution.

Summary

One very noticeable thing is that the VMs seem quite snappy and responsive. A lot more so than any of the other virtualization products I’ve tried on this server. And I get the impression that people are starting to use KVM for production workloads.  Sure I was disappointed that I couldn’t get Windows 7 to run, but thats not a big deal for me. I found the virt-manager was good at the basic ‘wizard to create a simple VM’, but some of the finer points about how it works have a way to go.