I’ve been setting up an ESXi system for a client recently, and I must admit there is a lot to like about ESXi. It pretty much works like it says in the brochure 😉 The client system is a proper Dell server type system and all the hardware is supported on ESXi (3.5u4 in this case). Of course, I’d like to play around with ESXi at home, but I don’t really have any hardware that supports it except my old Thinkpad T42. The better choice would be my core 2 duo tower system but it really needs an Intel gigabit LAN card to be of any use with ESXi, as the onboard Marvell (yukon2) LAN chipset is not supported by ESXi. (NB: The Marvell LAN I have is a 88E8053 on an old Asus P5LD2 motherboard)
However, there are a few projects on the net to port drivers to ESXi since the driver layer is based on some older linux kernels as far as I can tell. I found the open-vdrivers project on sourceforge that has some modules for some Realtek gigabit LAN cards for using with ESXi 3.5. There is also some info on the open-vdrivers site re setting up the build environment to compile the drivers for ESXi 3.5. Apparently you need an old gcc compiler (3.2 or 3.3 I think), and the recommendation was to use Centos 3 as a platform. It appeared as though they had taken the realtek driver source from a 2.6.18 kernel and modified it to get it to go in the VMware build environment.
So I thought I’d attempt to port the linux sky2 driver (what the Marvell Yukon2 LAN chipsets use) compiled and running on ESXi 3.5.
How hard could it be? (I must admit I’m a hackerish C programmer at the best of times and didn’t have high hopes).
Amazingly I got it to work. I honestly don’t know how well it works, but I’ve performed backups over the network with it and it hasn’t died.
So I installed Centos 3.9 i386 in a virtual machine, ticking the boxes to make sure I had the dev environment stuff installed (I ended up needing all three install CDs). I also needed subversion to check out the open-vdrivers project and I must admit trying to find subversion for Centos 3 was painful (I think I ended up finding a Subversion 1.2 rpm somewhere that worked). It’s probably easier to check out open-vdrivers on a more recent distro and then just copy the directory tree into your Centos 3 VM.
The notes on setting up the build enviornment for open-vdrivers is here at the moment. I’ll repeat a fair chunk of this below. You need the open sourced files from Vmware to get started. I was using 3.5u3 initially (since that was the only install iso I had), but later repeated the process with 3.5u4. All the notes below will just refer to 3.5u4. And yes, yes, I know 4i is out, but I haven’t even attempted to look at 4i yet.
So go to the VMware open source page and download the 3.5u4 OSS source code. The file I ended up with was VMware-eesx-public-source-3.5.0-153875.tar.gz. Extract that file in your Centos3 machine and you should end up with a bunch of other tar.gz files;
VMware-eesx-gpl-public-source-3.5.0-153875.tar.gz VMware-esx-server-public-source-3.5.0-153875.tar.gz VMware-esx-drivers-public-source-3.5.0-153875.tar.gz
Extract all those in a temporary directory. You’ll end up with a few more tgz files which aren’t that important as well as two directories;
VMware-esx-drivers-public-source-3.5.0-153875 VMware-esx-public-source-3.5.0-153875
Check out the open-vdrivers project inside the same temp dir (or check it out on another machine and then transfer the ‘open-vdrivers’ directory to your centos3 machine.
svn co https://open-vdrivers.svn.sourceforge.net/svnroot/open-vdrivers/trunk open-vdrivers
First you have to add a symlink into the VMware source distribution. In my case;
cd VMware-esx-drivers-public-source-3.5.0-153875/src/include ln -s ../../../VMware-esx-public-source-3.5.0-153875/include/lkhdrs/asm-i386
Now cd into the VMware-esx-drivers-public-source-3.5.0-153875 directory inside your temp directory and execute the build script for VMware’s own open sourced drivers;
cd ~/tmp/VMware-esx-drivers-public-source-3.5.0-153875 ./build-vmkdrivers.sh
You’ll get a fair few warnings and mine always ends in an errors about the usb-storage driver (see below). I don’t really care about that, and I don’t think it really matters for what you’re about to do.
**** Creating library drivers-usb-storage-obj.a ar: build/release/drivers/usb/storage/debug.o: No such file or directory ld: cannot open build/release/drivers/usb/storage/bin/drivers-usb-storage-obj.a: No such file or directory
Now we need to do a test compile of the open-vdrivers code. First you need to set up another symlink;
cd ~/tmp/open-vdrivers/src ln -s ../../VMware-esx-drivers-public-source-3.5.0-153875/src/include
And then try compiling one of the realtek drivers to check that it succeeds;
./build-r8169.sh ** Compiling drivers/net/r8169/r8169.c ** Compiling drivers/net/r8169/linux_module_heap.c **** Creating library drivers-net-r8169-obj.a
The actual module is in build/release/drivers/net/r8169/bin/
ls -l build/release/drivers/net/r8169/bin/r8169.o -rw-r--r-- 1 root root 979049 Aug 14 23:01 build/release/drivers/net/r8169/bin/r8169.o
OK, so far so good. Now, if you just want to try compiling my sky2 build, grab my sky2-for-esxi35u4-0.01.tar.gz and extract it into the open-vdrivers directory, then compile it;
./build-sky2.sh
And then you should end up with the driver in build/release/drivers/net/sky2/bin;
-rw-r--r-- 1 root root 975909 Aug 14 23:15 sky2.o
Yeah, I’m not so sure why its that big either. To get it to work on ESXi, I find it much easier if ESXi is on a USB stick. There’s a few howtos on the net for how to do this. Most seem to be Windows-centric. Here’s my brief linux take on it;
cd /tmp mkdir vmiso mount -o loop VMware-VMvisor-InstallerCD-3.5.0_Update_4-153875.i386.iso /tmp/vmiso mkdir /tmp/install cd /tmp/install tar xvzf /tmp/vmiso/install.tgz cd usr/lib/vmware/installer # Insert a USB stick of at least 1GB in size. Make sure it doesn't mount. If your linux distro auto # mounts it then unmount it. Run dmesg to find out what its device name is. In my case it was sdd, # so I put of=/dev/sdd below. This will vary depending on your machine. MAKE SURE YOU GET IT # RIGHT, otherwise you'll trash one of your harddisks) bunzip2 -c VMware-VMvisor-big-3.5.0_Update_4-153875.i386.dd.bz2 | dd of=/dev/sdd bs=32k
That bunzip2/dd thing will take a while. Wait until your USB stick activity LED stops flashing and then probably pull it out and stick it back in again. If you do a dmesg again you might see some partiion info;
[560078.250013] sdd: sdd1 < sdd5 sdd6 sdd7 sdd8 > sdd4
So that gets you a default install. Now how do we insert the sky2 driver into it? You need to firstly get the sky2.o driver onto the USB stick and then you need to make sure the PCI id information is updated so that it loads the correct driver at boot time. If you’re currently running linux on the machine you want to use for ESXi, you can run lspci to work out the PCI id for your card. On mine;
# lspci -nn ... 02:00.0 Ethernet controller [0200]: Marvell Technology Group Ltd. 88E8053 PCI-E Gigabit Ethernet Controller [11ab:4362] (rev 19) ...
The 11ab:4362 bit is my PCI id. You need to write this down.
There are few possibilities for which tar.gz file to change on the USB stick, but I used the oem.tgz method. In my case I had already downloaded the Unified oem.tgz from acronymlabs . This oem.tgz is actually for ESXi 3.5u3, but it works well enough for me on u4. Extract that oem.tgz file into a temp directory on your linux box,
cd /tmp mkdir oemstuff cd oemstuff tar xvzf ~/CommunityUnifiedDriverPack_v1.1.0_U3-123629.oem.tgz
then edit the etc/vmware/simple.map file and add an entry in for your PCI id. In my case, I added the bolded entry below;
1166:024b 0000:0000 storage sata_svw
1166:0411 0000:0000 storage ide
11ab:4362 0000:0000 network sky2
13c1:1003 0000:0000 storage 3w_9690
Now copy the build/release/drivers/net/sky2/bin/sky2.o file from your build directory into the mod directory
And create a new tar.gz file and make sure its called oem.tgz
cd /tmp/oemstuff tar cvzf /tmp/oem.tgz *
Insert the USB stick and mount partition 5 (or the one with the label Hypervisor1)
mkdir /tmp/usb mount LABEL=Hypervisor1 /tmp/mnt
You’ll see that there already should be a tiny oem.tgz file in that partition. You can back it up if you like, but there is pretty much nothing in it. So just copy your new oem.tgz file to the USB stick
cp /tmp/oem.tgz /tmp/mnt
And unmount it
umount /tmp/mnt
Now try booting off the USB stick (you might need to change your BIOS settings for USB booting, or I ended up pressing F8 to get the boot select menu most of the time). I disconnected all my linux hard disks before rebooting as a precaution. You can boot off the USB stick with no physical hard drives in your system, and on this first boot we just want to confirm that the sky2 card is recognised.
So you should see the ESXi boot up messages and some progress info about loading drivers. If you see ‘lvmdriver failed to load’, then that usually means your network card driver didn’t load. Once all the boot up stuff has finished you should get the status screen with an IP address onscreen (probably 0.0.0.0 initially). Now you can press Alt-F1 and type the word ‘unsupported’ (you won’t see any of these letters as you type it). Then you’ll get a root password prompt. just hit ENTER as you probably haven’t assigned a root password yet. Now you should have # root prompt. Have a look at the /var/log/config.log file;
less /var/log/config.log
Scroll through the file, and find where its loading the sky2 driver.Hopefully you should see something like;
Using /mod/sky2.o Module load of sky2 succeeded
If you don’t, or you see messages about unresolved symbols then it hasn’t worked. If it has worked, then you can ‘exit’ and press Alt-F2 to go back to the main status screen. Press F2 to configure ESXi. Assuming the network card is working, you should be able to configure an IP address for the mgmt interface as well as a hostname. Go to a machine running Windows, overcome your depression, and point a browser at https://<the IP you assigned to your ESXi box> and you’ll get a cert warning and then a default page that allows you to download the Infrastructure client from ESXi itself. Install that and configure your ESXi box.
For reference, I started out using the sky2 driver from a 2.6.18 kernel (since the realtek drivers seem to use this version as a base), but I wasn’t making a lot of progress, so then I read that ESXi 3.5 is closer to a 2.4.21 kernel, so I used the most recent 2.4 kernel (2.4.37) and grabbed the sky2 driver from it. That turned out to be a lot quicker to get to the point where it actually compiled successfully (NB: the final ar in the build script will always fail unless the ‘bin’ directory exists for the output file). Once I had it successfully compiled, it still failed to work in ESXi. Looking at the /var/log/config.log I was getting a few unresolved symbols alerts when it tried to load the sky2 module. I noted down the names of these symbols, then went back to my centos3 build system and searched a vanilla 2.4.37 kernel tree looking for these missing symbols. Fortunately they tended to be simple functions that had no further dependencies, so I basically just copied the code from the kernel directly into the driver and recompiled, put it all on the USB stick again and rebooted until the module loaded cleanly … and then amazingly it worked. I also tried hacking at the 3c59x driver in order to get a 3com 3c905 boomerang card to work (which seems to be the one type of 3c905 that is not supported by ESXi). Again I got it to compile and load, but honestly I don’t think it works properly. I had it working sort of OK as a secondary interface, but if ESXi decided to use it as vmnic0 (ie. the mgmt interface nic) then ESXi would crash on boot.
And finally a big disclaimer. I am certainly not a kernel driver programmer by any stretch. I somehow got this driver to compile and amazingly it works fine on my home network in the limited testing I’ve done. I still think for production use, go buy some intel pro 1000 LAN cards or get a motherboard with an intel gigabit lan integrated.
UPDATE: 2010/03/27. On my ‘Using a Marvell LAN card with Vmware ESXi 4’ post there’s been some discussion about using a different type of Marvell card; a 88E8001 which uses the linux skge driver. I’ve finally had some feedback that my latest attempt at an skge driver might be working. Thats under ESXi4. So I’ve had an attempt at doing much the same hackery of the linux 2.4.x skge drive for ESXi3.5 . So if you have a 88E8001 and want to try it with ESXi 3.5, grab skge-for-esxi3.5u4-0.04.tar.gz and post some feedback as to whether it works or not. You’ll need to edit the simple.map file and add an entry like;
11ab:4320 0000:0000 network skge.o
If the module does not work, please post any ‘unresolved symbol’ info you find in the logs. As per the other post, I don’t own an 88E8001 based device so cannot test it myself. This v0.04 release adds some suggestions by ‘Sheepless’ (see comments), but I am dubious that it will actually work. As per Sheepless’ comments, this skge driver might be a bit harder to do for ESXi 3.5. But, if you’re able to try it, please do, and post some feedback.
Having an old PC with an 88E8001, I tried your skge module with ESXi 3.5, but you still have some unresolved symbols:
Warning: unresolved symbol skb_headroom
Warning: unresolved symbol skb_padto
Warning: unresolved symbol skb_copy_expand
Warning: unresolved symbol skb_tailroom
Warning: unresolved symbol jiffies_to_msecs
Warning: unresolved symbol msecs_to_jiffies
The time conversion ones are easy to fix: you rolled the functions into skge.c with changed names, but forgot to change some references to the original names. Unfortunately, the other problems are tougher. I started merging the relevant sk_buff-related stuff into the driver, but soon ran up against compilation problems, and a comment in VMware-esx-public-source/include/linux_skbuff.h shows why: “This is the small subset of sk_buff stuff we actually use in the vmkernel linux emulation layer”. Basically, they’ve made changes to struct sk_buff, so changes are needed to the driver code. I’ve had a crack at this, but it’s still panicking, and I’ve got a feeling there are deeper problems.
Thanks for trying. Thanks for listing the unresolved symbols. I’ll have a closer look when I get some time.
Been following the ESXi 4.0 article and am definitely very interested in this, as my target ESX server can only do 32-bit. I have a DGE-530T NIC so I can definitely help in testing the driver. I am proficient in C and Ethernet as well if needed (but not in Linux drivers).
Sheepless, I see what you mean. I had a go at fixing those unresolved symbols, and I can get it to compile, but I suspect my compile may have the same panics as yours. Please grab the skge-for-esxi3.5u4-0.03.tar.gz tar file and see if works any better. All the changes I’ve made are just more routines (out of a 2.4.37 kernel) jammed into the top of the skge.c file.
daymz, if you feel like trying the latest incantation of the skge driver (see comment above), please do. I suspect it won’t work, but see how it goes.
Oops, Am pretty sure that 0.03 version won’t work. Will probably get an unresolved symbol about copy_skb_header.
OK, try this 0.04 version ; skge-for-esxi3.5u4-0.04.tar.gz. Note, for Sheepless, that copy_skb_header thing highlights just how different the sk_buff struct is now. I just commented out a heap of stuff and shoved it into skge.c. Again, probably won’t work, but worth a try.
Thanks for this blog. I had bought a Dlink DGE-528T gbit NIC which was suppose to be a Realtek 8169S card. I thought that it was going to be supported out-of-the-box by the customized oem.tgz file. But it did not work, even after I had included the PCI ID into the simple.map file. I then started pulling source codes from various sites to compile as per your instructions above but faced a lot of challenges esp the skb_ stuff.
One thing I did to make things easier was to dd the installation into a USB stick. Then I kept changing the oem.tgz file and just boot it on the PC without the card install. I can then use the vmkload_mod command to try and load each module.o file to see if there are unresolved symbols in the message log. So you don’t actually need the card to test for unresolved symbols.
In the end, I check the source in the open-vdriver for the r8169.c file and I found that actually the PCI IDs are ALSO SET in the module source code (in a struct named pci_tbl) and if your card’s id is not in there then it will NEVER detect the card even if you specified it in the simple.map file.
So I made changes to the source, recompiled it and now the DGE-528T is recognized with the driver r8169 and it works.
There is another way to make the change which is to do a binary replace of one of the other pre-defined pci-id in that file to fill in your PCI ID, using hexedit, I think.
Hence, if you have problems in getting your card recognized and you are sure that it has a supported chipset, consider making the PCI ID change in the module file.
BTW, I am also getting pretty large object files and I don’t know how to shrink / reduce / minimize its size.
Also I have a problem fixing the unresolved symbol kmap_skb_ and kunmap_skb_ as per your skge-for-esxi3.5u4-0.04.tar.gz source above. Did you managed to overcome these references ?
cheelee, thanks for the comment. I never tried doing that vmkload_mod for the drivers I’ve compiled where I don’t have the physical card/chipset. I always assumed that it would never say anything useful. I might have to have another look at that skge driver to try that out. For that skge driver, we seem to get it working for ESXi 4, but not for ESXi 3.5.
Thanks for the tips about the PCI id.
Re the large object files, the build scripts I use tend to have plenty of debug turned on by default. Try editing your build…sh script and editing the CFLAGS line so that it does not include the ‘-g3’ bit. Then recompile. For the ESXi 4 build scripts, there are two variables at the top of the scripts; DEBUG and DASHG. Just comment those line out.
As for that skge-for-esxi3.5u4-0.04.tar.gz, I assumed there might me some unresolved stuff, but I didn’t get any feedback about it. Like I said I might have another go now using the vmkload_mod, as per your suggestion.
Hi, I am building a driver for the Realtek 8139 chipset using the 8139too.c source from 2.4.18 kernel. I just found out that you need to include 2 new lines into the driver for it to be recognized by the ESX kernel. You need to include:
SET_MODULE_OWNER(dev);
SET_NETDEV_DEV(dev, &pdev->dev);
after the “alloc_etherdev()” function. Otherwise, the esxcfg-nics command will not see the NIC card and it cannot be configured.