Using a Marvell LAN card with ESXi 4
Well, after somehow getting my Marvell LAN card working with ESXi 3.5u4 (and u3) I thought I’d have a look at ESXi 4. Again I somehow got it to go. I’m not too sure how good it works, but it works well enough for me at home. If you can’t be bothered reading about me going on and on and on and on about how to compile it, then just scroll to the bottom of the post. The download for the source includes a precompiled module.
ESX/ESXi 4 is quite different from 3.5. The build chain is similar to 64 bit Redhat/Centos 5.2, so I ended up installing a x86_64 Centos 5.3 inside a vmware fusion machine to do my dev work. I just made sure I installed all the dev stuff. Then I downloaded the VMware-esx-public-source-4.0-162945.tar.gz from VMware’s open source page. It’s a much bigger file (590MB) than the file for 3.5. When you extract the file you end up with a lot of rpm files plus a vmkdrivers-gpl.tgz file. I did the following to extract it all on my test machine;
cd ~ mkdir vmware-oss cd vmware-oss tar xvzf ~/VMware-esx-public-source-4.0-162945.tar.gz mkdir drivers cd drivers tar xvzf ../vmkdrivers-gpl.tgz
One of the rpm files included is a kernel source rpm. I’m not exactly sure what it is relevant to ESX, but I installed it anyway for reference. I found I needed the qt-devel and gtk2-devel packages first;
cd ~ cd vmware-oss yum install qt-devel yum install gtk2-devel rpm -iv kernel-sourcecode-400.2.6.18-128.1.1.0.4.159770.x86_64.rpm
I’m pretty sure you don’t need the kernel source to build the drivers, but I kept it handy for reference anyway.
You can try doing a test build of the drivers now. This will build all the drivers built in to ESX/ESXi.
cd ~/vmware-oss/drivers ./build-vmkdrivers.sh
You’ll probably get a few warnings, but it should complete. If you do a find down the ‘bora’ directory you should see a bunch of .o files corresponding to the kernel modules (look under the ‘bora/build/scons/build’ directory).
OK, my approach was to look at the build-vmkdrivers.sh script and basically look at what was done to compile one network driver (I used the forcedeth driver as a reference) and just make a reduced script for my sky2 driver. As for the source to base the sky2 driver on, instead of using a driver from 2.4.37 like I did with the 3.5 version of the network driver, this time I ended up using the sky2 source from 2.6.26 (or the debian lenny incantation of it). I did originally use the sky2 driver from the kernel-sourcecode…2.6.18…rpm file, but on closer inspection of the tg3 driver that ESX uses, I noticed it actually comes from a 2.6.24.1 kernel (or thereabouts) … so I thought I may as well use a more modern reference source. I had a few minor hiccups trying to get it to compile, but in the end I just had the following shoved into the top of my sky2.c file;
/* Stuff for ESX compile */
#define upper_32_bits(n) ((u32)(((n) >> 16) >> 16))
#define csum_offset csum
#define bool int
#define PTR_ALIGN(p, a) ((typeof(p))ALIGN((unsigned long)(p), (a)))
u32 bitreverse(u32 x)
{
x = (x >> 16) | (x << 16);
x = (x >> 8 & 0x00ff00ff) | (x << 8 & 0xff00ff00);
x = (x >> 4 & 0x0f0f0f0f) | (x << 4 & 0xf0f0f0f0);
x = (x >> 2 & 0x33333333) | (x << 2 & 0xcccccccc);
x = (x >> 1 & 0x55555555) | (x << 1 & 0xaaaaaaaa);
return x;
}
The define's are to remedy compilation errors, and the bitreverse is to satisfy an undefined symbol problem. Note that the undefined symbols errors end up in /var/log/messages on your ESXi 4 box now (unlike 3.5).
So in the end to compile my driver I did a;
./build-sky2.sh
You get a couple of warnings, but if you do a 'find . -name sky2.o' you should end up with two sky2.o files. There is a DEBUG and DASHG variable defined at the top of the build script. If you uncomment these it'll build a lot of debug stuff into the modules.
If you get some errors, its probably because some directories are missing in the build path, so make them first;
mkdir -p bora/build/scons/build/vmkdriver-sky2.o/release/vmkernel64/SUBDIRS/vmkdrivers/src26/drivers/net/sky2 mkdir -p bora/build/scons/build/vmkdriver-sky2.o/release/vmkernel64/SUBDIRS/vmkdrivers/src26/common/
Now, again I used a USB stick with ESXi 4 (build 171294 in my case). To install it, I loopback mounted the VMware ESXi 4 iso file file, extracted the image.tgz file to a temp directory, bunzip2'd the big dd image file, then dd'd it to the whole USB stick.
A difference this time is that I didn't have a simple.map file all pre-prepared to go into the oem.tgz file. I thought the easiest way to get it would be to just boot ESXi and let it fail when it tries to configure a network device, then somehow copy the simple.map file off. So I did this. ESXi 4 merrily boots and eventually you see the dreaded 'lvmdriver failed' message. It looks like ESXi is broken at that point, but just type the word 'unsupported' and you get a password prompt, and just hit ENTER to get a prompt.
Because networking is not working, we'll just copy the simple.map to the Hypervisor1 partition;
cp /etc/vmware/simple.map /vmfs/volumes/Hypervisor1
I just did a 'sync' and held in the power switch ( perhaps type 'reboot' if you feel like being more careful). Now get the USB stick to appear as a USB device in your development VM (your centos 5.x environment), and mount partition 5 (or the Hypervisor1 partition) off the USB drive, and you should see an oem.tgz file as well as the simple.map file. You need to make a directory structure up for the new oem.tgz file we'll be creating;
cd ~ mkdir vmtest cd vmtest mkdir -p etc/vmware mkdir -p usr/lib/vmware/vmkmod
Copy the simple.map off the USB drive into etc/vmware directory in our tree structure. eg.
cp /mnt/simple.map ~/vmtest/etc/vmware
And edit the simple.map so that it includes the PCI ids for your Marvell card. Mine is 11ab:4362 so I added in the bolded line below, but yours could likely be different. If you're not sure, you could boot ESXi off the USB stick again, do the 'unsupported' thing to get a prompt and type lspci -v
1166:0410 0000:0000 storage sata_svw.o
1166:0411 0000:0000 storage sata_svw.o
11ab:4362 0000:0000 network sky2.o
14e4:1600 0000:0000 network tg3.o
Now copy in the sky2.o file that we compiled earlier. The modules are in a different directory compared to 3.5 (NB: the compilation process produces two sky2.o files, so make sure you grab the one shown below)
cd ~/vmware-oss/drivers cp ./bora/build/scons/build/vmkdriver-sky2.o/release/vmkernel64/sky2.o ~/vmtest/usr/lib/vmware/vmkmod
Now tar it up, and copy it to the USB stick that should be still mounted;
cd ~/vmtest tar cvzf ~/oem.tzg * cp ../oem.tgz /mnt
Unmount the USB stick
umount /mnt
Now try booting again. Hopefully you should see a 'loading sky2' flash up early in the boot ... and it should eventually get to the usual ESX status screen showing the current mgmt IP address. Basically if you don't see the 'lvmdriver load failed' then here's a good change it's working.
And yes here is the sky2-for-esxi4-0.01.tar.gz download. It includes the build script, the modified source, plus directory tree for creating the oem.tgz file including a pre-compiled copy of the module. If you can't be bothered compiling, you can just extract this file, cd to the vmtest directory and create the oem.tgz file as per the earlier notes.
UPDATE: (2010/02/08) There is also now a driver for the Marvell 88E8001 LAN chipset (see the comments discussion below). This uses the skge driver, not the sky2 driver mentioned above. I don't own a 88E8001, so thank you to samarium for helping out re the unresolved symbols. Please comment below if you've tried the skge driver and it works. I have a new tarball sky2-and-skge-for-esxi4-0.02.tar.gz containing both the sky2 and skge driver
Hi there!
Firstly, a very well written blog entry.
I too have a Marvell Yukon and wanted to evaluate ESX/(i)4.
The installation isn’t proceeding beyond the unrelated complaint about not being able to load the lvmdriver.
I’m not installing from CD/DVD or USB but via PXE (obviosuly replacing the root oem.tgz file).
My particular chipset is a 88E8001, so slightly different to yours.
I’d hoped merely changing the ’simple.map’ file to reflect my vendor/product ID would work (I changed mine from your 11ab:4362 to my 11ab:4320).
Upon reboot I noticed the Sky2.o module being loaded (it flashed by) but then presents me with the same lvmdriver loading error message (I have a 64GB SATA SSD and a 500GB SATA 2.5″ HDs present in a Shuttle SN95G5 (nForce3 chipset)).
Is it possible to use a precompiled Marvell .ko file from somewhere similar to the kernel 2.6 one here; http://www.marvell.com/drivers/driverDisplay.do?driverId=153?
If so; does this need to be compiled against a working install with all of the compile flags as per your source/build script?
Any ideas as I’m loathed to fill the only free PCI slot with an Intel e1000 (as I want to use it for a nice new TV-Tuner for a HTPC)?!
Looking forward eagerly in anticipation of a response.
Cheers,
John
PS: Don’t suppose you know if it’s also possible to dual boot ESX4/(i) and XenServer5.5 (and if so; know a good HOWTO)?
Wow, someone actually reads this!
Get into ‘unsupported’ mode (ie. alt-f1, type ‘unsupported’ and maybe a password) and cd /var/log and do a ;
less sysboot.log
and scroll until you find the messages about loading sky2. Mine just says ;
Module sky2 loaded successfully.
When trying to get the driver to work I tended to see a lot of ‘unresolved symbol’ warnings in the sysboot.log when it attempted to load sky2. Do you get any of these? A possibility with your onboard Marvell is that its referencing different parts of the driver to mine … hence the possibility of unresolved symbols.
If you run a regular linux distro on this host, do you know which network module gets loaded. Is it definitely sky2? or maybe the skge one? (maybe boot a linux ISO and get to a prompt and try lsmod |grep sk to see which one is loaded). I just had a go at getting skge to compile (using the driver from 2.6.26), and it seemed to compile ok. I have no idea if it will work, but you can grab it from here. Of course you’ll need to shove it in your oem.tgz and change the simple.map so that it references skge.o, not sky2.o. Tell me how you get on. (UPDATE: 2010/1/25. Looks like the skge driver obviously does not work. It was a ‘best guess’ compile. But it’s very difficult to diagnose what the problem is without a machine with an skge card in it. As per the comments below, if someone can post what the actual unresolved symbols are, that will be a ’start’ in working out how to resolve it).
[...] If you want to go straight to ESXi 4.0, KernelCrash has you covered there as well. [...]
Hi,
I have Marvell 88E8001 card (11ab:4320) and can’t get it workwith ESXi 4.
Tried sky2.o driver… Driver loaded successfuly but card not recognized by ESXi.
Tried skge.o driver… and log says: “vmkload_mod: Can not load module skge: Unresolved symbol”
CentOS 5.4 Live CD loads skge module and card works fine.
There are pretty new drivers ond Marvell site but I don’t know how to compile them to work with ESXi 4.
Regards
Not too sure. Is there anything else listed in the sysboot.log?. Like if you set it up to use just the skge.o driver, is there anything else in the sysboot.log? Often it lists exactly what the unresolved symbols are. Alternatively, if you configure it to boot using just the sky2.o driver instead, what does ‘esxcfg-vmknic -l’ return (in ‘unsupported’ mode)?
Hi Kernel,
I got 3 Marvell 88E8001 (11ab:4320) at home and like to use them for EXSi 3.5 U5. I tried the SKGE.O drivers you provided, I can’t get it work.
It did tried to load the SKGE.O but a lot of unresolved symbols in the /var/log/config.log
did you complied SKGE.O for ESXi 4.0 only? If yes, can you make a version for ESXi 3.5 (U5)? I saw another topic from you is about Marvel Yukon Nic on ESXi 3.5, it seems same as someone post on vm-help.com, I did try the SKY2.O no luck, as they mentioned in the source code of the SKY2.0 driver has no 11ab:4320.
Let me know if you need any more info from me.
Thanks in advance.
Kernel,
I got more info, based on the lspci -vvv output, the NIC was driven by SKGE under SLAX linux live CD. So I believe SKGE.O will be the better bet. I found Marvel’s latest driver for 88E8001 from here:
http://extranet.marvell.com/drivers/files/Linux_10.81.6.3.zip
As I got 2 old PC without 64BIT & VT-x, I can only install ESXi3.5 on them. One of them even has 3C940 on board NIC, seems not support neither.
Looking forward to hear from you, thanks
i’ve been trying to get the latter marvell network controller working with ESXi4 also, and had same conclusions as the previous poster.
according to this debian mailing list, the driver for the 11ab:4320 would be the sk98lin.
http://lists.debian.org/debian-boot/2004/02/msg00230.html
do you know if that driver is available to compile for esxi 4?
Thanks,
~tim
As per an earlier reply, for those with an 88E8001 based card, I need more information about what is happening when esxi is booting. Like I said; Is there anything else listed in the sysboot.log?. Like if you set it up to use just the skge.o driver, is there anything else in the sysboot.log? Often it lists exactly what the unresolved symbols are (basically if someone can tell me what the actual unresolved symbols are I can have another go at compiling it with these missing symbols compiled in). Without an actual 88E8001 card I can only really guess.
ESX4i U1 (build 208167) on an ASUS P5Q-e.
$ lspci | awk ‘/AHCI|Gigabit/’
00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
02:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 12)
07:02.0 Ethernet controller: Marvell Technology Group Ltd. 88E8001 Gigabit Ethernet Controller (rev 14)
$ lspci -n|egrep `lspci | awk ‘/AHCI|Gigabit/{l=l s $1; s=”|”}END{print l}’`
00:1f.2 0106: 8086:3a22
02:00.0 0200: 11ab:4364 (rev 12)
07:02.0 0200: 11ab:4320 (rev 14)
ACHI driver for ICH10R supported out of the box.
sky2 driver provided by KernelCrash works in oem.tgz but with 11ab:4364 id, thanks.
skge driver failed to load. skge is what normally runs on this box under ubuntu, so not sure what previous commenter was talking about sk98lin? Maybe previous a driver for this card.
sysboot.log output from system trying to load skge:
vmkload_mod: Can not load module skge: Unresolved symbol
[2010-02-06 08:55:29 'VmkCtl' warning] Loading module skge.o failed. Exec of command ‘/sbin/vmkload_mod skge ‘ succeeded, but returned with non-zero status: 1
messages output from system trying to load skge:
Feb 6 08:55:29 vmkernel: 0:00:00:16.456 cpu2:4717)Loading module skge …
Feb 6 08:55:29 vmkernel: 0:00:00:16.456 cpu2:4717)Elf: 2320: symbols tagged as
Feb 6 08:55:29 vmkernel: 0:00:00:16.464 cpu2:4717)WARNING: Elf: 1570: Relocation of symbol failed: Unresolved symbol
Feb 6 08:55:29 vmkernel: 0:00:00:16.471 cpu2:4717)ALERT: Elf: 2518: Kernel module skge was loaded, but has no signature attached
Feb 6 08:55:29 vmkernel: 0:00:00:16.471 cpu2:4717)WARNING: Elf: 2542: Kernel based module load of skge failed: Unresolved symbol
so it looks like skb_pad is the culprit.
Hey thanks for that. I guess the skb_pad bit is off the end of the lines you pasted in. I’ve found the skb_pad routine out of the same kernel I used for the skge driver (2.6.26) and shoved it into the driver source code, and recompiled. I’ve uploaded this new test version of the skge driver. Can you tell me if it works .. or perhaps gets other symbol errors?
I’ll have a look later today.
Looking at the comment I left compared to what I pasted, I thnk the line got chopped by wordpress, because what it said on the Elf: 1570 line between symbol and failed was
lessthan skb_pad greaterthan
so I guess wordpad in interpreting it as a unknown html token.
Now skge loads, and works in so far as I can ping test successfully. Thanks.
Not sure if it will be useful to me, but since you went to the trouble of building it, I thought it would be nice to get it tested.
If I get keen, I’ll build up my own build environment, and maybe take a crack at getting the dual port PCI-E Silicon Image 3132 adapters I have working under ESXi, but too many other projects on the go for it to happen soon.
Thanks again for the NIC drivers.
Hey, thats great news. It’d be good to see whether it still works OK under some load. I might update the main post re the skge driver working now and encourage people to test it more thoroughly and comment. Thanks again.
kernel,
I’m trying to create and add drivers to a ESXi 4 install. Have any tips: http://www.vm-help.com/forum/viewtopic.php?f=12&t=2002
Thanks!
I don’t know much about that Broadcom card. If I was trying to get it to go, I’d work out whether there is an existing linux kernel driver for that Broadcom chipset, and use that rather than a linux driver off the broadcom site. The rationale is that a built-in linux kernel network driver is probably going to be more like the sky2 or skge drivers that I’ve modified OR some of the drivers that vmware themselves supply the source for (such as the forcedeth one). You can easily see the changes I made to say the sky2 linux driver by just downloading the linux kernel source for 2.6.26, pull out the sky2.c file and diff it against the one that I’ve included as a download here (or the skge.c one). I didn’t end up making many changes. Just have a go at trying to get your driver to compile. Once it compiles OK, then you might find some unresolved symbols popping up in the esxi boot logs, fix all those and see if it works.