Using a Marvell LAN card with ESXi 4

(Note: This post was initially written when ESXi 4.0 was available. As of late 2010, ESXi 4.1 has been released, and it does actually include a sky2 driver that may or may not work with various Marvell LAN chipsets. The post is still relevant (especially the comments)  if your particular Marvell chipset does not work with the sky2 driver in ESXI 4.1. Also, the post is relevant if you’re interested in porting other network drivers to ESXi)

Well, after somehow getting my Marvell LAN card working with ESXi 3.5u4 (and u3) I thought I’d have a look at  ESXi 4. Again I somehow got it to go. I’m not too sure how good it works, but it works well enough for me at home. If you can’t be bothered reading about me going on and on and on and on about how to compile it, then just scroll to the bottom of the post. The download for the source includes a precompiled module (NB: As per the post about ESXi 3.5, this is all about getting a 88E8053 chipset Marvell LAN working).

ESX/ESXi 4 is quite different from 3.5. The build chain is similar to 64 bit Redhat/Centos 5.2, so I ended up installing a x86_64 Centos 5.3 inside a vmware fusion machine to do my dev work. I just made sure I installed all the dev stuff.  Then I downloaded the VMware-esx-public-source-4.0-162945.tar.gz from VMware’s open source page. It’s a much bigger file (590MB) than the file for 3.5. When you extract the file you end up with a lot of rpm files plus a vmkdrivers-gpl.tgz file. I did the following to extract it all on my test machine;

cd ~
mkdir vmware-oss
cd vmware-oss
tar xvzf ~/VMware-esx-public-source-4.0-162945.tar.gz
mkdir drivers
cd drivers
tar xvzf ../vmkdrivers-gpl.tgz

One of the rpm files included is a kernel source rpm. I’m not exactly sure what it is relevant to ESX, but I installed it anyway for reference. I found I needed the qt-devel and gtk2-devel packages first;

cd ~
cd vmware-oss
yum install qt-devel
yum install gtk2-devel
rpm -iv kernel-sourcecode-400.2.6.18-128.1.1.0.4.159770.x86_64.rpm

I’m pretty sure you don’t need the kernel source to build the drivers, but I kept it handy for reference anyway.

You can try doing a test build of the drivers now. This will build all the drivers built in to ESX/ESXi.

cd ~/vmware-oss/drivers
./build-vmkdrivers.sh

You’ll probably get a few warnings, but it should complete. If you do a find down the ‘bora’ directory you should see a bunch of .o files corresponding to the kernel modules (look under the ‘bora/build/scons/build’ directory).

OK, my approach was to look at the build-vmkdrivers.sh script and basically look at what was done to compile one network driver (I used the forcedeth driver as a reference) and just make a reduced script for my sky2 driver.  As for the source to base the sky2 driver on, instead of using a driver from 2.4.37 like I did with the 3.5 version of the network driver, this time I ended up using the sky2 source from 2.6.26 (or the debian lenny incantation of it). I did originally use the sky2 driver from the kernel-sourcecode…2.6.18…rpm file, but on closer inspection of the tg3 driver that ESX uses, I noticed it actually comes from a 2.6.24.1 kernel (or thereabouts) … so I thought I may as well use a more modern reference source. I had a few minor hiccups trying to get it to compile, but in the end I just had the following shoved into the top of my sky2.c file;

/* Stuff for ESX compile */
#define upper_32_bits(n) ((u32)(((n) >> 16) >> 16))
#define csum_offset csum
#define bool int
#define PTR_ALIGN(p, a)         ((typeof(p))ALIGN((unsigned long)(p), (a)))

u32 bitreverse(u32 x)
{
        x = (x >> 16) | (x << 16);
        x = (x >> 8 & 0x00ff00ff) | (x << 8 & 0xff00ff00);
        x = (x >> 4 & 0x0f0f0f0f) | (x << 4 & 0xf0f0f0f0);
        x = (x >> 2 & 0x33333333) | (x << 2 & 0xcccccccc);
        x = (x >> 1 & 0x55555555) | (x << 1 & 0xaaaaaaaa);
        return x;
}

The define’s are to remedy compilation errors, and the bitreverse is to satisfy an undefined symbol problem. Note that the undefined symbols errors end up in /var/log/messages on your ESXi 4 box now (unlike 3.5).

So in the end to compile my driver I did a;

./build-sky2.sh

You get a couple of warnings, but if you do a ‘find . -name sky2.o’ you should end up with two sky2.o files. There is a DEBUG and DASHG variable defined at the top of the build script. If you uncomment these it’ll build a lot of debug stuff into the modules.

If you get some errors, its probably because some directories are missing in the build path, so make them first;

mkdir -p bora/build/scons/build/vmkdriver-sky2.o/release/vmkernel64/SUBDIRS/vmkdrivers/src26/drivers/net/sky2
mkdir -p bora/build/scons/build/vmkdriver-sky2.o/release/vmkernel64/SUBDIRS/vmkdrivers/src26/common/

Now, again I used a USB stick with ESXi 4 (build 171294 in my case). To install it, I loopback mounted the VMware ESXi 4 iso file file, extracted the image.tgz file to a temp directory, bunzip2’d the big dd image file, then dd’d it to the whole USB stick (NB: There are some details about how to do this in linux on my other post re ESXi 3.5. See link at the top of the post)

A difference this time is that I didn’t have a simple.map file all pre-prepared to go into the oem.tgz file. I thought the easiest way to get it would be to just boot ESXi and let it fail when it tries to configure a network device, then somehow copy the simple.map file off. So I did this. ESXi 4 merrily boots and eventually you see the dreaded ‘lvmdriver failed’ message. It looks like ESXi is broken at that point, but just type the word ‘unsupported’ and you get a password prompt, and just hit ENTER to get a prompt (You might need to hit alt-f1 first before typing ‘unsupported’)

Because networking is not working, we’ll just copy the simple.map to the Hypervisor1 partition;

cp /etc/vmware/simple.map /vmfs/volumes/Hypervisor1

I just did a ‘sync’ and held in  the power switch ( perhaps type ‘reboot’ if you feel like being more careful). Now get the USB stick to appear as a USB device in your development VM (your centos 5.x environment), and mount partition 5 (or the Hypervisor1 partition) off the USB drive, and you should see an oem.tgz file as well as the simple.map file. You need to make a directory structure up for the new oem.tgz file we’ll be creating;

cd ~
mkdir vmtest
cd vmtest
mkdir -p etc/vmware
mkdir -p usr/lib/vmware/vmkmod

Copy the simple.map off the USB drive into etc/vmware directory in our tree structure. eg.

cp /mnt/simple.map ~/vmtest/etc/vmware

And edit the simple.map so that it includes the PCI ids for your Marvell card. Mine is 11ab:4362 so I added in the bolded line below, but yours could likely be different. If you’re not sure, you could boot ESXi off the USB stick again, do the ‘unsupported’ thing to get a prompt and type lspci -v

1166:0410 0000:0000 storage sata_svw.o
1166:0411 0000:0000 storage sata_svw.o
11ab:4362 0000:0000 network sky2.o
14e4:1600 0000:0000 network tg3.o

Now copy in the sky2.o file that we compiled earlier. The modules are in a different directory compared to 3.5 (NB: the compilation process produces two sky2.o files, so make sure you grab the one shown below)

cd ~/vmware-oss/drivers
cp ./bora/build/scons/build/vmkdriver-sky2.o/release/vmkernel64/sky2.o ~/vmtest/usr/lib/vmware/vmkmod

Now tar it up, and copy it to the USB stick that should be still mounted;

cd ~/vmtest
tar cvzf ~/oem.tgz *
cp ../oem.tgz /mnt

Unmount the USB stick

umount /mnt

Now try booting again. Hopefully you should see a ‘loading sky2’ flash up early in the boot … and it should eventually get to the usual ESX status screen showing the current mgmt IP address. Basically if you don’t see the ‘lvmdriver load failed’ then there’s a good chance it’s working.

And yes here is the sky2-for-esxi4-0.01.tar.gz download. It includes the build script, the modified source, plus directory tree for creating the oem.tgz file including a pre-compiled copy of the module. If you can’t be bothered compiling, you can just extract this file, cd to the vmtest directory and create the oem.tgz file as per the earlier notes.

UPDATE: (2010/02/08) There is also now a driver for the Marvell 88E8001 LAN chipset (see the comments discussion below). This uses the skge driver, not the sky2 driver mentioned above. I don’t own a 88E8001, so thank you to samarium for helping out re the unresolved symbols. Please comment below if you’ve tried the skge driver and it works. I have a new tarball sky2-and-skge-for-esxi4-0.02.tar.gz containing both the sky2 and skge driver

UPDATE: (2010/12/24). xieliwei has a driver that should support the 88E8057 on ESXi 4.1 (as per the latest comments, it seems a bit iffy on 4.0). Anyone else who has this chipset, if you could try the driver and post more information that would be great. The 88E8057 code is here as a local copy ; sky2-1.22-for-esxi4-88e8057-r1-xieliwei.tar.gz or on mediafire; sky2-1.22-for-esxi4-88e8057-r1-xieliwei.tar.gz . Also there’s another version of the driver that will produce copious debug information (only of use if you’re having problems); http://www.mediafire.com/file/m836gce3r3b7ygi/sky2-1.22-for-esxi4-88e8057-r1-oem-debug-xieliwei.tgz

118 Responses to “Using a Marvell LAN card with ESXi 4”

  1. well after a few more minutes, pings are errratic again.

  2. Maybe its something to do with the esxi version you’re using, are you able to update to 4.1?

  3. yes, I will download 4.1 tonight and check it out. Thanks for the hard work.

  4. After switchimg to 4.1 the sky2 driver appears to work at least for management on my 88e8057.

  5. garyfloyd, does that mean that you are still experiencing problems?

    I did a install of XP last night using the vmnet bridged interface to the 88e8057 and ran windows update overnight, it completed okay.

    I wonder what has changed between 4.0 and 4.1, sounds like it was almost working in 4.0 though. For others who wish to try the driver on 4.0, here is a oem.tgz for my debug version of the driver, just rename to oem.tgz:
    http://www.mediafire.com/file/m836gce3r3b7ygi/sky2-1.22-for-esxi4-88e8057-r1-oem-debug-xieliwei.tgz

    By debug I mean lots of amateurly placed kprintf statements. The vmkernel console will be spewing out status statements for every RX frame.

    I’m interested in what the statements are before and when the driver starts/is failing. If anyone wishes to try, please copy the messages.* files from /var/log out and provide me a copy.

    Take note that if the driver eventually goes into an endless loop, the messages.* files will be overwritten when the total log size reaches about 7MB, please copy the logs out before then.

    Kernel’s method of obtaining the simple.map file from a running system is a good method for getting files off ESX with a non-functioning NIC.

  6. the driver appears to be working corectly with 4.1. however, that is just based on pings and the vsphere client. I have not used that interface for actual vm traffic yet.

  7. it works also for me. (esxi 4.1 on marvell/galileo 11ab:4380 – mod.88e8057) pc is lenovo thinkcentre 7303-wp5.
    Tested iscsi with openfiler for a couple of hours continuos disk access with hdspeed.exe on a vm! AWESOME! 🙂
    one question:
    can i do think trick on ESX 4.1 ?thx!
    from italy with love, an happy sysamin!

  8. Yay! =)

    Is that question directed to me? I’m not sure what think trick is. Assuming you mean ‘this’ instead of ‘think’, I cannot guarantee that it would work on ESX since I don’t have an ESX license. However, since ESXi is just the core version of ESX, I don’t see why it wouldn’t work.

    Just a heads up after my 3 days worth of testing; it appears that the driver can PSOD when disabling the interface from the yellow management interface. I couldn’t replicate it but it did happen twice, both involving the sky2_poll() function. My hunch is a race condition bug brought in from the original driver. It shouldn’t be a big problem though as one normally wouldn’t disable an interface.

  9. I think ESX and ESXi are fundamentally the same thing at the basic kernel level that we’re looking at here

  10. I don’t see it mentioned in the comments above, but VMware snuck in a sky2 into 4.1. I’ve posted some details here – http://www.vm-help.com/esx41/sky2_driver.php.

  11. Thanks Dave. I still need to get around to looking at 4.1. So does the installer include the sky2 driver, or is it just the source code included in the oss files?

  12. I seem to recall seeing a sky2.o module the first time I tried getting the 88e8057 working, but I was lacking sleep at that time… Somehow after modifying the simple.map file to load that module, it reported missing symbols.

  13. I just upgraded my Dell T110 server from ESXi 4.0 to 4.1. The intention was to get a new Galileo NIC working as a secondary port, and I read that the sky2 driver was present in 4.1. This Galileo uses the Marvell 88E8075 (11ab:4370).

    After performing the simple.map and pci.ids customizations, I could get the machine to recognize the card as something it could passthru, but not as a NIC for the host. Looking in the logs revealed the same undefined symbol problem with bitreverse as described above.

    I used the driver from sky2-1.22-for-esxi4-88e8057-r1-xieliwei.tar.gz straight up using the same oem.tgz mechanism, and the host was able to recognize it right off. Very cool. Thanks a million!

  14. Just tested xieliweis driver on a Shuttle SX58J3 with 88E8057 onboard.
    management and simple VM Network seem to work smoothly, thanks a lot!!!

  15. […] digging around revealed the steps necessary to create this homebrewed version.  The author provides a link to a precompiled version, but this doesn’t (yet) have the […]

  16. hi samarium

    I am also using Asus P5Q-E

    The below are what I have found
    marvell 88e8056
    pci id: 11ab:4364

    marvell 88e8001
    pci id: 11ab:4320

    When I load sky_2.o the was discovered successfully.

    however when I put in skge.o together. Instead marvell 88e8056 was not found but marvell 88e8001 was loaded. I discover the screen with red text stating

    0:00:00:23.538 cpu2:4808)Elf: 3043: Kernel module skge was loaded, but has no signature attached.

    Were you able to load both drivers?

  17. If I were to load just skge.o alone, I would also encounter
    0:00:00:23.538 cpu2:4808)Elf: 3043: Kernel module skge was loaded, but has no signature attached.

    marvell 88e8001 will still be load.

    It seems like I cannot get to have both Marvell nic running. There is some signature error on skge.o

    I have tried both version

  18. Fantastic! I was able to get the integrated Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller NIC (11ab:4364) working in my Shuttle SG31 (SG31G20) with the Sky2 driver posted here, and after some editing of pci.ids and simple.map.

Leave a Reply

You must be logged in to post a comment.