So I’ve been a big virtual machine user for many years. Mainly using the desktop products like Vmware Workstation (and Fusion on the Mac) and I’ve tried Virtualbox a few times. I tried VMware Server a few times but was never happy with its performance on linux. But, Vmware ESX is always quite impressive. Being so low level it obviously has quite a few extra ’smarts’ over the desktop and ‘on top of a regular OS’ type VM products. It is incredibly popular in data centers, and has the key benefit of ‘being around for years’, and has consequently been refined and refined.

So that brings me to KVM. In theory it should be the linux world alternative to ESX. Sure Xen kind of does the same thing, but KVM is very integrated into the kernel supposedly, whereas Xen has its own microkernel or somesuch. Of course, when KVM first came out, I tried it and it didn’t seem to work very well, so I forgot about it until now. Lately, I’ve been looking at other distros and have been playing quite a bit with Ubuntu 9.04. Usually I would hate something like Ubuntu due to the volume of updates you typically get, but I was interested in ext4 performance and I had also read that KVM was reasonably integrated into it, so it seemed like an easier way to try those things out.

So I installed Ubuntu 9.04 (the 32 bit version initially) on my 2Gb ram core2duo on top of ext4 and read this KVM post at Adoleo to get some hints on setting it up. I must admit there is a lot of ‘not quite’ consistent information on the net about setting up KVM … and I can understand people who have tried setting it up, got stuck, googled for answer … and then given up. The Adoleo post is quite good as it’s quite recent. There were a couple of extra apt-get’s I needed to do to get everything going (you need the qemu package for example).

Now, with all the bits installed, I of course wanted to install a guest. This is where I think a lot of info on the net becomes confusing. It’s very much a case of heaps and heaps of ‘abstractions’. Here’s how I think everything fits together;

• kvm itself is a low level kernel module that does the virtualization stuff using the Intel VT extension or the AMD SVM extension
• qemu is a software  i386 (and others) CPU emulation system that has been around for years and has a lot of tools necessary for doing virtualisation stuff. In the past qemu has been slow.
• qemu now can leverage itself against kvm to do ‘in hardware’ virtualisation. So I think if you run the kvm command (which seems ot take all the same args as the qemu command) on a VT or SVM capable CPU and you just happen to be emulating a intel type PC architecture system then it’ll go and use the dandy fast hardware virtualization stuff.
• Launching these virtual machines using the kvm command at the command line means you end up with a long list of args. Someone must have thought ‘Lets come up with some clever XML abstracted network thingee instead’ and hence you have this libvirt API which seems to be an API for managing virtual machines both locally and across a network. All the args to the kvm command are now in the form of some XML file.
• Someone thought we need some GUI management tool just like vmware etc have and you end up with virt-manager which seems to be used in the majority of examples on the net. virt-manager uses the libvirt API.

Well, all the above is how I think it works … today

So based on that Adoleo info for the basic low level setup, I think I started up virt-manager (you might have to apt-get it too) and you get a window that says;

localhost (System)

localhost (User)

Of course I could not find a howto anywhere that mentioned why there are two entries and what each one is for. What I think they mean is that localhost (System) is for all the virtual machines running as root level , and the localhost (User) is for all the machines running as your username (???). Anyway I clicked the User one and ‘connected’ to it OK, then right clicked to create a new VM. I think I tried a ubuntu 9.04 VM first. That sort of worked OK. You get a dandy graphical console view and even though the mouse was sometimes erratic (there is a hint somewhere to configure the virtual mouse as a usb tablet to get around this) I got the VM installed OK.

Then of course I could not get networking to go. I’ve set up bridging using brctl before so I had an idea in my head how I wanted things to work. I changed my /etc/network/interfaces to bridge my eth0 into a br0 bridge (as per that Adoleo page), and my host side networking worked fine. Guest networking was a no go. it eventually clicked that even though my user had write permission to /dev/net/tun that it probably was not creating the tap devices in the first place etc. I googled for a while, gave up, then thought “I’ll just make the VM’s part of that localhost (system) thing which should run as root and get rid of my network probs.

This meant working out where the libvirt XML config files live for the VMs. They live in ~/.libvirt/qemu. Just mv them from there into /etc/libvirt/qemu, and restart the libvirt daemon (I seem to have to do this restart a lot);

/etc/init.d/libvirt-bin stop

/etc/init.d/libvirt-bin start

Probably shutdown all your VMs too and stop virt-manager and start it again and you should now see all your VMs under ‘localhost (System)’.

Now my networking worked. I had something like this in my XML file for the ubuntu VM;

<interface type=’bridge’>
<mac address=’54:52:00:12:34:56′/>
<source bridge=’br0′/>
<model type=’virtio’/>
</interface>

and also I had the following somewhere for the mouse;

<input type=’tablet’ bus=’usb’/>

So now I tried a Windows XP VM. It also worked. I had some fun with the networking. Initially I used a line like the ones above, which specify that virtio thing. Virtio is some special driver interface that KVM virtual machines can use to get much faster disk and network performance. It requires a special driver in the guest OS. I think in the case of ubuntu 9.04 this is already integrated. With my XP VM I would just get a ‘Ethernet Controller’ device detected … and for the life of me I could not track down where the Windows XP virtio drivers now live on the net. All the links to them on the net seem either broken or really old. So I kind of gave up, and my XP VM now just has a network section like so (without the virtio ref);

<interface type=’bridge’>
<mac address=’54:52:00:65:43:21′/>
<source bridge=’br0′/>
</interface>

In the XP device manager I now just see a Realtek 8139C network device. It is autodetected and configured and networking just works.

So then I tried a Windows 7 RC VM. The only ISO I have is for the 64 bit version. Of course, one of the limitations of KVM is that you cannot run a 64 bit guest on top of a 32 bit OS (I think some of the vmware products can do this). Anyway, my host OS install was relatively new (and destroyable) so I downloaded the 64 bit iso of Ubuntu 9.04 desktop and got that up and running. And set everything up AGAIN.  Now I tried the Windows 7 install and suffice to say it does not work. It starts, it goes a bit of the way through the install and then dies. So far I’ve googled but haven’t found a solution.

Summary

One very noticeable thing is that the VMs seem quite snappy and responsive. A lot more so than any of the other virtualization products I’ve tried on this server. And I get the impression that people are starting to use KVM for production workloads.  Sure I was disappointed that I couldn’t get Windows 7 to run, but thats not a big deal for me. I found the virt-manager was good at the basic ‘wizard to create a simple VM’, but some of the finer points about how it works have a way to go.

Recently I had a problem with a Debian Etch server I maintain for a client. The possible solution to my problem involved downloading and installing a new program. Of course, this new program seemed to want a lot of dependencies met. I started to apt-get these and when apt-get told me it might need to install 100MB+ of related packages, the alarm bells went off and I hit ctrl-c. A better solution would be to try this on a test Etch server … but I didn’t have one.

OpenVZ to the rescue. Basically I downloaded a Debian Etch template for OpenVZ, installed the program and its dependencies and basically worked out the solution to my problem.Sure you can do the same thing with other virtualisation technologies but the key advantages to me were just how quick it was to do everything;

Firstly the setup/install process is quick;

• The example templates on the OpenVZ site are all relatively small (the Etch one is 112MB). A lot smaller than a CD or DVD install image.
• To install the image takes a few seconds
• I already had my notes for the handful of commands to create a VM, so I had the VM running and on an IP address on my LAN in a few minutes
• Since the debian build is minimal, an apt-get update/upgrade is also very quick.

Another key factor that is not obvious is just how fast you can start and stop and restart a VM. There’s no hardware probing as the kernel boots since each VM has no kernel. The overhead is the startup time of the relevant startup scripts. In my test VMs it’s in the order of a few seconds to startup or shutdown. This is obviously handy when you’re testing something and you need to reboot several times.

For some background, the big problem with ESXi is that I didn’t think I had anything that was compatible. It has a very limited range of hardware it works with. I did try it on my Core2Duo desktop server when it first came out, but it couldn’t find a disk to install on, and I had low hopes of a Marvell gigabit card ever working anyway. There is the highly useful VMware ESX Whitebox HCL. That pretty much confirmed that my ASUS motherboard was never going to work and I needed an Intel Pro 100 or 1000 network card (and intel Pro 1000 cards seem to be particularly expensive in my part of the world).

So I tried installing it on my T42. Of course the installer failed, saying it cannot find a disk it can write to. It seems that ESXi is aimed at people with SCSI and SATA  or iSCSI drives and the lowly T42 can only do IDE. However, there is a way to set it up on the internal IDE disk (NB: As per this outie.net howto on the IDE ESXi trick , you need to press Alt-F1 to flip back to the first console screen after you type ‘install’. I thought this was rather confusing). Using this technique, ESXi installs just fine. You  reboot and the network interface works and you have a very compact ESXi server

OK, so ESXi installs. Now what?

This next bit turned out to be rather painful. The main way of administering your shiny new ESXi server is to use Vmware Infrastructure Client which only works on Windows! Doh! So I thought I’ll run up Vmware Fusion on my Mac, install Vmware Infrastructure into my XP VM and do it all from there. This sounded like a great idea, but I came across an annoying problem related to running a VMware product within a VMware product.

VMware Fusion gets you to press Ctrl-Alt to release the mouse and keyboard back to the host machine. Guess what Vmware infrastructure gets you to press to release its mouse/keyboard back to the host machine? Yes, ctrl-alt as well. So, in Vmware Infrastructure I create a test virtual machine to boot off, but I had trouble getting it to pick up a virtual CDROM, so I would press ctrl-alt to release the mouse/keyboard, but sadly Fusion picked this up and I found my mouse/keyboard were back at the ‘Mac’ level. It seemed rather difficult to get back to the Windows desktop (ie. I could not do things like press the ‘Stop’ or ‘Restart’ buttons in Vmware infrastructure).

One workaround I did find was to go to the ESXi server and restart the Management Agents … which then made Vmware Infrastructure pop up a message saying it’d lost contact with the server.

Eventually I found a setting in the settings for my virtual machine to set a boot delay. I set this to 7000ms and that gave me time to click the connect CD/DVD button and find the Fedora iso I was testing this with …. and eventually it booted.

Fedora must use a vmware X driver in its installed as I no longer had my ctrl-alt problem once it installed

Another problem I had was when I tried to create a second VM and got a message in Vmware Infrastructure like ‘Admission Check failed for Memory Resource’. I eventually found this post about reducing the VIM memory . There’s a post part way down that page about changing the VIM object reservation from 512MB to 192MB. That did the trick for me.

Installing a Windows XP Pro VM requires getting a SCSI driver floppy image (I couldn’t see how you could use an IDE disk in a VM). I ended up downloading the floppy image from http://download3.vmware.com/software/vmscsi-1.2.0.4.flp . For the Windows install, first I had to click the connect CD/DVD and then after you see the CD booting, click the ‘connect floppy’ and select the image. Then when the installer prompts you to press F6 for any additional drivers, it should find the driver floppy image.

So it generally  worked. But performance was pretty crap. It’s probably an OK test set up to get to know ESXi and the Infrastructure client … but I probably wouldn’t use it to run actual useful servers on.

However, setting it up on the core2duo box was just a test. I really wanted to set it up on my old Thinkpad T42. It’s been gathering dust since I bought the Macbook, and it seemed such a shame to have it doing nothing. I’ve had an old Thinkpad 600X running Asterisk for a while now. It runs natively (ie. not in a VM) and does the job, but it has a hand compiled kernel and Asterisk … and that kind of set up tends to put me off doing proper updates on it. Hence why I was liking Debian Lenny; modern kernel and Asterisk 1.4.x in the standard repository. And the T42 had a bit of horsepower with a 1GB of RAM and a 100GB drive, so I thought something like Asterisk could be virtualised on it.

So step 1, install Lenny from CD on the T42 and ‘apt-get update’ , ‘apt-get upgrade’, then step 2 ; install the openvz kernel image;

apt-get install linux-image-2.6.26-1-openvz-686

And reboot … and nothing. It would not boot. Obviously my Thinkpad did not like this kernel. It turns out that the Debian Lenny OpenVZ kernel is compiled with PAE support compiled in. A core2duo has PAE support. The Pentium M in the Thinkpad does not support PAE.

So what to do? I could go on about the other virtualisation techniques I tried here … but ultimately I liked the idea of OpenVZ; Simple kernel mod, light overhead, don’t need dedicated raw files on disk for VMs. I also liked that Lenny had a standard OpenVZ kernel in it’s repository, which ultimately meant ‘easy kernel updates’.

So I thought I’d have  a go at trying to recompile the Debian OpenVZ kernel with PAE turned off … in a supportable way. To me the easy way is to get a vanilla kernel from kernel.org, and add the patches from openvz.org. But I wanted to have a kernel closely approximating the latest apt-get’able Debian OpenVZ kernel.

I’ve used make-kpkg to make my own debian kernels before, but that didn’t really work in this case.

So here’s what I did (I got a lot of ideas from this post about Xen without PAE);

# get the source
cd /usr/src
apt-get source linux-image-2.6.26-1-openvz-686

Now you should end up with a /usr/src/linux-2.6-2.6.26 directory. It looks like a normal kernel extraction directory, but with a debian directory.

cd /usr/src/linux-2.6-2.6.26/debian/config/i386
vi config.686-bigmem

Now I just searched for PAE in the partial config file and turned it off, as well as disabled the 64GB setting and enabled the 4GB one. Mine looks like this:

CONFIG_HIGHMEM4G=y
CONFIG_HIGHMEM64G=n
## end choice
CONFIG_X86_PAE=n

Save that. Now do all the steps to apply patches, build the config, and compile to generate the kernel image and headers deb

cd /usr/src/linux-2.6-2.6.26
fakeroot debian/rules debian/build debian/stamps

# Removing these ABI files below is a cheap hack workaround to errors that are normally produced at the end of the compile

rm debian/abi/2.6.26-1/*
fakeroot make -f debian/rules.gen binary-arch_i386_openvz_686

That last fakeroot command will take ten bazillion years to complete. Once complete you should end up with two deb files in /usr/src;

linux-headers-2.6.26-1-openvz-686_2.6.26-13_i386.deb
linux-image-2.6.26-1-openvz-686_2.6.26-13_i386.deb

Now install both of these;

cd /usr/src
dpkg -i linux-headers-2.6.26-1-openvz-686_2.6.26-13_i386.deb
dpkg -i linux-image-2.6.26-1-openvz-686_2.6.26-13_i386.deb

And make sure you have the OpenVZ tools installed;

apt-get install vzctl

And try rebooting

This time it worked for me.

And now (finally) we set up a virtual machine and install Asterisk in it.

Download an OpenVZ template for Lenny from http://openvz.org/download/template/cache;

debian-5.0-i386-minimal.tar.gz

You just put the tar.gz file in /var/lib/vz/template/cache (ie. don’t extract it).

Then its a case of picking a unique id for your VM (eg. 101) and doing  something like;

vzctl create 101 –ostemplate debian-5.0-i386-minimal –config vps.basic
vzctl set 101 –onboot yes –save
vzctl set 101 –hostname testmachine –save
vzctl set 101 –ipadd 10.1.2.3 –save
vzctl set 101 –nameserver 1.2.3.4 –save
vzctl start 101

The VM should start quickly and to get a sort of console you enter;

vzctl enter 101

One thing that confused me initially is how networking works. The basic example above uses venet networking. I wasnt sure whether the IP address I specified should be on a seperate subnet or on the same subnet as my host machine. In the end, I used an IP address that was on the same subnet as my host machine. It doesn’t use brctl style bridging, but it effectively bridges the interface onto your network. I must admit I couldn’t find a decent explanation/diagram of how OpenVZ networking works.

Install asterisk in the VM;

apt-get install asterisk

The install worked fine, but because of problems with Music On Hold in the past I wanted to set up ztdummy properly. This can be done.

In the host machine I installed the zaptel modules per this debian howto

apt-get install zaptel-source
module-assistant prepare
module-assistant auto-install zaptel

Afer you do that, do a ‘modprobe ztdummy’. The first time I tried this it didn’t work, because Lenny had downloaded the ‘official’ package for the OpenVZ kernel headers rather than the one that I generated. If that happens to you, dpkg -r the old package, and then install the headers deb that you created earlier. Then run the auto-install above again and double check that the ztdummy modules inserts cleanly. If you do a dmesg now, you should see something like;

ztdummy: High resolution Timer started, good to go

So that gets ztdummy working from the host machine. We need to make it accessible from within the virtual machine. I looked at the following two howtos;

http://wiki.openvz.org/Asterisk_in_VE_with_debian_stable
http://wiki.openvz.org/Asterisk_from_source (I just used the vzctl example commands they have)

The big test of whether the ztdummy stuff is working from inside the virtual machine is to run zttest and hit ctrl-c after a few lines of output. Here’s how mine runs;

root@lenny:/# zttest
Opened pseudo zap interface, measuring accuracy…
99.996582% 99.975983% 99.984177% 99.983887% 99.984070% 99.984467% 99.984375%
99.984177% 99.979492% 99.984467% 99.981735% 99.983208% 99.984581% 99.982819% 99.983978%
99.982811% 99.977448% 99.983971% 99.983971% ^C
— Results after 19 passes —
Best: 99.997 — Worst: 99.976 — Average: 99.983484, Difference: 99.983484

So the only bit remaining for me was to configure asterisk. I just copied files over from my old Asterisk server and got it going. The only issues I had;

• I needed to use the g729 codec from http://asterisk.hosting.lv/. That just requires dropping the library file in the right directory and restarting asterisk.

• I found I had to change the asterisk.conf file so that it had the following change;astdatadir => /usr/share/asterisk
  Debian seems to put all the sounds and music on hold data under /usr/share/asterisk (instead of /var/lib/asterisk). I had some ulaw and g729 sounds on the old server, so I copied those into /usr/share/asterisk/sounds

And that is pretty much it. What an endeavour. In hindsight, if I had counted the cost of all the effort put in to get this working on a non-PAE CPU, it would have been easier to just buy a newer system with a PAE CPU.

UPDATE: I used the simple venet networking when I initially set up the Asterisk virtual machine. Unfortunately, the VM would work for a few hours and then either lose its default route or be unable to get responses to DNS requests … so I ended up converting the VM to veth networking using the notes here and here. It now works much better.

UPDATE: I noticed after doing this, that an apt-get update/upgrade may end up trying to download a kernel with the same name as the one I compiled. There should be a way around this, but for now I’m doing the following to prevent my kernel and headers being overwritten (NB: You may have to undo the change below in order to install a new self-compiled kernel. You probably just need to change ‘hold’ to ‘install’ to undo the change below);

# echo “linux-headers-2.6.26-1-openvz-686 hold” | dpkg –set-selections
# echo “linux-image-2.6.26-1-openvz-686 hold” |dpkg –set-selections

UPDATE: I had problems with DNS lookups occasionally hanging from within the OpenVZ virtual machine. If I did a ‘cat /proc/user_beancounters’ from inside the virtual machine, the failcnt column would have a non-zero number against the dgramrcvbuf.  Apparently this has something to do with udp buffers. As a quick workaround I did this to double the barrier limit for that parameter;

sudo vzctl set 102 --dgramrcvbuf $((262144*2)) --save

alias snd-card-0 snd-hda-intel
options snd-hda-intel index=0
alias snd-card-1 snd_pcsp
options snd_pcsp index=1

It seemed to fix it.  Then earlier today I was trying to play a youtube video and could barely hear it. I’d been playing a video in mplayer earlier and the volume level seemed to be fine. So I checked the Gnome Volume control in the top right corner and it was at 100%, then I ran alsamixer and the main volume and the PCM one were almost at 100 as well. Then I checked the volume control in the flash window for the youtube video. It was full. Then I double checked the actual volume of the speakers on my monitor. They were pretty high too.

This got me thinking about linux sound in general; Sound on linux has been a pain in the ass for quite a while now. I know it works ‘most of the time’ … but it’s one of those things that you’d kind of like to work ‘all the time’. Why on earth should anyone even have to consider about 4 volume controls? That is obviously silly.

So per usual I started doing some research, and came across ‘The Sorry State of Sound in Linux’ . It’s over a year old, but I think it’s probably still quite relevant. It is sort of a mini history of how linux sound ended up in the mess that it has. An interesting read. I must admit I have never really played with aRts and ESD or pulseaudio or any of those software mixing things as I generally run things using ALSA directly.

There is a subtle hint in that blog post to try out what was the closed source OSS (by 4Front technologies). Its now open source as far as I can tell … and you can just download it and install it for free. So I did. I just grabbed the linux 2.6 x86 deb and installed it (which was very very easy). Rebooted. It detected my Intel HDA onboard sound and then I ran osstest (which makes nice-ish test noises rather than highly annoying ones). In Gnome, the volume control applet in the top right corner no longer works (not entirely sure why), but I just run ossxmix to change the volume. It seems to work fine. And like a few of people noted on that blog post … the sound output seems to sound better than anything ALSA produced.

So, so far so good. I can even play that youtube video now and hear it.

There’s some interesting comments in that blogpost. This one sounds a chord with me;

“This type of BS is exactly why many of us long time linux users have given up and bought macs, or … windows.”

I’m not suggesting you buy a Mac or Windows PC. However, if like me you’ve used Linux since the 90’s you may be starting to tire of some of these issues that seem to sound more like politics rather than real technical issues.

Problem one was that on maybe 80% of occasions (ie. not all the time), just after startup my program would crash with a ‘glibc double free’ error. It spat out a stack trace that wasn’t much use to me. I’d never seen these double free errors before so I assumed they are something to do with the new glibc used .. OR could be some debug stuff that is in Lenny (since technically it hasn’t gone stable yet … its just a release candidate).

So I google for ‘glibc double free’ … and well you can try yourself … but google returns a zillion results all of people having these errors…. with not a lot of solutions offered. I did work out that it means that a program tried to do a free() on a block of memory twice. I summarily checked all my free() calls and didn’t see anything bad. I then played with gdb a bit … which did’t really help. My program is multi-threaded and I’ve never really worked out how to use gdb to effectively debug such a program (I should probably learn). So then I added lots of printfs to try to narrow down where the crash occurs (and keep in mind the crash doesn’t happen all the time).

So it seemed to start one or two threads and then die shortly after the 3rd thread started. All these download threads run identical code. The bug (I think) related to some openSSL initialisation code early in the thread. My download threads can make either clear text socket connections to a server … or SSL based ones. I had some lines similar to:

connection.method=SSLv23_client_method();
connection.context = SSL_CTX_new(connection.method);

But these were being run regardless of whether an SSL connection was going to be established. I added an if statement around these to only run them for SSL connections and voila … no more glibc errors …. well none that I’ve seen.

Weird eh? I did find there is an env variable you can set to turn off all this double free checking;

export MALLOC_CHECK=0

Problem two was that the unrar program provided with Lenny is a fair bit newer than the one in Etch. My program behaves like ‘expect’ forking unrar and watching its output and prompts and responding appropriately. Of course, the text output of the newer unrar (3.80 beta I think) is slightly different to the one in Etch. Suffice to say I had to rewrite some of my ‘expect’ code and finally it works again.

However, recently I tried to upgrade the non-free adobe flashplayer plugin to 10.something … and hey it doesn’t work on Etch. Basically the flashplayer needs a more modern glibc (as to why glibc keeps having so many major version changes is beyond me). Anyway, I started reading about the next version of Debian ; Lenny. They currently have it as an RC1 so I thought that must be reasonably stable. So I downloaded the netinst ISO for the 64 bit version. My current Etch is 32bit, and I’ve never seen much necessity to upgrade to 64 bit, as I don’t have lots of memory and I’ve never seen a review that says ‘the 64 bit version goes much better than the 32 bit one’.

So I installed the 64bit lenny on a separate disk to play with for a while. It all seemed good. Didn’t seen any faster really … but I did notice (at least for the 64 bit Lenny) that the flash player is the free one (can’t remember its name). I remember reading that adobe had recently released a 64 bit alpha version of the flashplayer for linux … but the fact that it was alpha was a worry.

I then remembered that debian lets you do a whole distribution upgrade and I had never tried it. So I made a dd backup of my root partition and gave it a shot. It’s relatively simple to do (google for ‘debian etch lenny dist–upgrade’):

• edit /etc/apt/sources.list and change all the references to ‘etch’ to ‘lenny’.
• Do a ‘apt-get update’
• Do a ‘apt-get dist-upgrade’

And then wait a long time. It does download a lot … and then you’ll get asked some questions about various packages and also some questions about some files that have changed. The latter generally related to some printer drivers I had manually installed. I made a note of these files but told the installer to use the new versions (After the install was finished I manually loopback mounted my dd of the old root partition and compared by hand)

The main prob after the install was getting X windows to start. I have an nvidia video card and I had installed a linux driver using a driver direct from nvidia. I see now (maybe they’ve always been there) that you can install these nvidia drivers using apt-get. I can’t remember the exact steps, but it went something like;

• apt-get install nvidia-glx
• apt-get install nvidia-xconfig
• dpkg-reconfigure xserver -xorg

I suggest googling for more help. The nvidia-xconfig thing seems to be definately required. And when you do the reconfigure I told it not to use the framebuffer for mode switching … but I’m not sure if thats required or not

I also had problems printing afterwards. But that turned out to be a major ‘Doh!’ moment as my printer had no paper in it … and the LED indicators aren’t exactly very descriptive.

Speedwise, Lenny doesn’t seem much different. My gdm login possibly takes longer now … which I’m thinking may be more to do with data fragmentation due to the major upgrade. Not sure. I’ll see how it goes.

I have one program I’ve been working on lately that is getting more complicated. For most of it’s life it’s been ‘command line only’. It creates multiple threads , opens lots of socket connections, spawn’s external subprocesses too. It’s been quite a learning curve to get it to work. Lately, I’ve started adding features to it to open a GUI window to provide status info. My first attempt was to simply add in some Xlib calls (yes Xlib ) to basically do the bare minimum to get a window up. That worked OK on linux, but I’d already ported the command line version to run on OSX … and the Xlib version did not really work on OSX.

So I thought ‘How hard would it be to turn my program into a Cocoa program?’. Cocoa is of course the main GUI enviornment on the Mac. In my earlier porting experience, I aas simply treating OSX as ‘another unix’ to do the bare minimum to port my program at a command line level. Using Cocoa was a whole different ball game.

So I thought I’d have a go. It’s quite a steep learning curve going from ‘procedural C programmer using vi as a development environment’ to ‘Using Objective C with Cocoa on the XCode development environment’.

I of course tried to make a very simple first app using an XCode tutorial. That proved a problem as a lot of the tutorials on the net are for XCode 2.x (rather than 3.x that is out now), and Apple have changed the Interface Builder enough to make all the old tutorials very difficult to follow.

But I persevered. I ended up getting the book ‘Cocoa Programming for Mac OS X’ which is actually a very well written book.  That kind of gave me enough info to get some of the basics of Objective C (I still have a lot to learn).

After 2 days hacking at my code, I now have a working program. I thought it would take quite a bit longer. I still don’t have a clue when it comes to some of the philosophical arguments about object oriented programming … but ultimately I just want a ‘first pass’ that sort of works.  Memory management is probably rubbish, and I don’t fully understand how the event loop is meant to work .. and I know I have a delegate set up … but I don’t really understand what a delegate is.

I guess working in XCode was actually moderately fun. It at least told me about the 170 warnings in my program (I really do leave a lot of unused variables all over the place).  Debugging info was OK … but ultimately debugging multithreaded stuff is very difficult at the best of times.

But the xbox is getting old, and the main thing it can’t do is very high resolution stuff, especially high resolution H264 content.

I did look at using the bottom half of a Thinkpad T40 as a media centre, but that never worked out very well. So I thought I’d try the mac mini. I’ve had the mac mini about a year now using it in my office, but since I bought the macbook the mini gets less and less use. So, even though it seems a waste to sit the mini under the TV, I thought I’d try it for a while.

I also thought I’d upgrade the mini from Tiger to Leopard first. I thought I’d do a clean reformat install, so found a 200GB drive I had, put it in a USB enclosure, told the mac mini to do a Carbon Copy of its internal 80GB drive, and then went and installed Leopard.

My goal was to use the Migration Assistant to migrate data back …. but I put that on hold, and just set the thing up to run FrontRow. One thing that works much nicer in Leopard is NFS mounts, specifically automounts. I used to manually run a script in Tiger to mount NFS mounts as it never seemed to work the way i wanted. So in Leopard I just put some symlinks under ~/Movies to /net/<fileserver>/movies and it all just worked fine.

I discovered a program called the ‘Sapphire Browser’. It’s a plugin for FrontRow that I think was originally developed for the Apple TV but also works on Leopard FrontRow. Anyway, it sort of indexes your media collection and adds in coverart and movie/tv descriptions. It mostly works for me. I think it really doesn’t like my NFS mounts, as I find it very difficult to update the information. The whole concept is similar to the ‘Library’ functions in XBMC. It all looks pretty … but I always wonder whether it’s that useful from a useability perspecpective.

I’m connecting the mac mini to the TV with a DVI to HDMI cable, and audio is just out the headphones socket of the mini to the TV (which has left and right audio next to the HDMI input as an option). It works well enough. On my TV, the HDMI is forced to 1280×720, even though the native resolution of the TV panel is 1368×768, so when looking at text its not as crisp as perhaps the VGA input (I should try VGA for comparison). Quality wise I still think the component output of the xbox is slightly better. I can now play 720p stuff OK, though I do see the occasional glitch in playback of these. I need to investigate further whether this is because the data is via NFS or just an issue with FrontRow.

I’ve also added the Mini in to my Harmony remote. That took a while to get sorted. It works OK, but I haven’t worked out a way to simulate pressing the play/pause button for 5 seconds in order to make the mini go to sleep.

Anyway, I eventually got round to grabbing my Carbon Copy of the mac mini’s disk to take a look at (It’s a seagate 200GB IDE drive). I first tried connecting it to my macbook via the USB enclosure. It would not mount at all. I eventually plugged it into my linux server directly and it found lots of bad sectors on it. I tried Seagate’s seatools and it too found lots of bad sectors. Not good. How on earth could I lose two disk drives so close together? This one was in constant daily use until I bought that 750GB drive. Then it sat in one of those shipping case things on the floor for the last couple of weeks.

I ran some dodgy sector recovery type programs (I never know what these things are really doing) and they seemed to recover some sectors. And now I’ve just been running some deep surface scan type recovery thingee program (which seems to take a good 12hrs to run). It actually looks a lot better than it was. With my 300GB drive that died, it got progressively worse over a few days. This one seems to be getting better. Given that my USB enclosure cost $20, I really wonder if its a case of a very dodgy power supply.

Unmirrored root? Yeah, what was I thinking.

Of course in the midst of all my reboots, the box decided not to come up. I’d had a few kernel builds that failed to boot so I assumed it was my fault, but I started to see funny DMA errors on the drive that has the root fs on it. Oh no.

And the drive was making strange noises too. This is never a good sign.

So then I set about to save the root fs. The drive was not completely dead, so  I  plugged it into a USB enclosure and had my T42 running dd_rescue against it. This eventually got all of my root partiton (minus 120Kbtyes). I fsck’d it and it seems reasonable. I really need to copy it file by file somewhere as there are bound to be files with holes in them.

Next problem. So I go and buy a new drive (a 750GB Samsung SATA), but in the course of my troubleshooting and multiple boots off CD (debian etch disk 1), my RAID5 arrays get into a bit of a weird state. Now, I had partitioned these disks into 50GB chunks and then RAIDed each of those (there is a method to this madness which I won’t go into here). So I have md5 which gives 100GB useable (50 + 50 + another 50 for parity), md6 is set up in a similar way, then md7 through to md10.

The weird state is that with the two working drives left, md9 and md10 say they are inactive when I do a ‘cat /proc/mdstat’. If I do a ‘mdadm –detail /dev/md9′ I get a state of ‘Active, Degraded, Not started’. Trying a ‘mdadm –run /dev/md9′ gives me an IO error. I also tried a ‘mdadm –assemble –force /dev/md9 /dev/sda9 /dev/sdb9 missing’ , but that just tells me the array is already active.

I check the Event id on the two partitions left in md9 (mdadm -E /dev/sda9 … mdadm -E /dev/sdb9) and they give me the same number, so in the theory the array should come up with the two drives.

So I trawl the net, and end up partitioning up my replacement drive and trying to add it in. So I tried a ‘mdadm /dev/md9 -a /dev/sdc9′. That added the drive in, but no resyncing started, and md9 stayed in its useless state.

More trawling the net, and I tried:

echo clean > /sys/block/md9/md/array_state

Finally, the array started syncing.  So now I’ve gone through and added in the all the extra partitions I’ve set up on the new drive (eg.’mdadm /dev/md5 -a /dev/sdc5′, ‘mdadm /dev/md6 -a /dev/sdc6′ … and so on).

Did a vgscan to make sure I could see my volume group. Then a ‘modprobe dm-mod’ as I was still booted off the Debian Etch install CD here (I just booted it as though I was doing an install, then went to manual partitioning, and chose the Software RAID option. Then don’t do anything further in the debian installer, hit alt-f2 and get another shell running. Now I can do cat /proc/mdstat and other stuff from this other terminal).

I had to do a vgchange volgroupname -a y

vgdisplay -v should now show lots of logical volumes. Now I go through and e2fsck -f /dev/volgroup/logicalvol for each of the logical vols. This takes quite some time, but its a safety check for me to make sure it all appears to be OK (I realise e2fsck is not going to do a complete check). It all checks out OK.