In a Xen setup, the Xen hypervisor runs directly on the hardware (bare-metal). The first guest operating system (dom0) runs “on top” of Xen and has full access to the underlying hardware. Additional guests (domU) also run on top of the Xen hypervisor, but with limited access to the underlying hardware.

Converting an existing Ubuntu install to a Xen dom0 install requires installing the Xen hypervisor. In previous versions of Ubuntu this could easily be done using apt. Unfortunately no packages exist for Ubuntu 9.10 (Karmic Koala). In this post I’ll describe the steps I took to install Xen from source on Karmic. If you’d rather install via binary packages you’ll need to find a third-party repository.

Install Xen hypervisor and tools

Install Xen Kernel Sources

The stock Karmic kernel will not work when running as dom0 in Xen. You must install a new kernel. I chose to compile my own kernel. Compiling a kernel is not for the faint of heart, but isn’t as hard as you might think.

You have two main choices when it comes to a dom0 kernel. The Xen kernel or a dom0 pv-ops kernel. Normally I would recommend using a pv-ops kernel. The same pv-ops kernel can run on bare-metal or under the Xen hypervisor. The pv-ops kernel is going to be the default in future versions of Xen and will most likely be included in the mainline Linux kernel soon.

Unfortunately, the pv-ops kernel will not work with binary graphics drivers provided by Nvidia. Since I have an Nvidia graphics card (and want to use the binary drivers) I need to use the standard Xen kernel. The standard Xen kernel is still version 2.6.18, however luckily Andrew Lyon maintains forward ported patches for Gentoo that we can use on our Ubuntu install.

First things first, let’s get the kernel sources downloaded and patched:

At this point I like to modify the kernel’s “extra version” to clearly identify that this is a dom0 kernel. To do this, modify the Makefile using your favorite text editor and change the EXTRAVERSION line to look like:

VERSION = 2
PATCHLEVEL = 6
SUBLEVEL = 31
EXTRAVERSION = .8-xen0-amd64
NAME = Man-Eating Seals of Antiquity
...

Compile Xen Kernel

Configuring your kernel is left as an exercise for the reader. There is a ton of existing documentation on how to do this. The easiest way to get started is to copy the configuration from the stock Ubuntu kernel and run make menuconfig.

There are some options that you want to be sure you enable:

Processor type and features --->
  [*] Enable Xen compatible kernel
Networking support --->
  Networking options --->
    <*> 802.1d Ethernet Bridging
Device Drivers --->
  XEN --->
    [*] Privileged Guest (domain 0)
    <M> Backend driver support
    <M>   Block-device backend driver
    <M>   Block-device tap backend driver
    <M>   Block-device tap backend driver 2
    <M>   Network-device backend driver
    <M>   PCI-device backend driver
    <M>   SCSI backend driver
    <M>   USB backend driver
    <*> Block-device frontend driver
    <*> Network-device frontend driver
    <M> SCSI frontend driver
    <M> USB frontend driver
    <*> Framebuffer-device frontend driver
    <*>   Keyboard-device frontend driver
    [*] Disable serial port drivers
    <*> Export Xen attributes in sysfs
        Xen version compatibility (3.3.0 and later) --->
           3.3.0 and later

Now compile and install your kernel:

If you compiled the Xen net, block and usb backend drivers as modules, you will probably want to load them by default by adding them to your /etc/modules file:

# /etc/modules
...
# These modules are required for Xen DomUs to work
netbk
blkbk
usbbk

Update Grub

Most existing Xen documentation on the web refers to the first version of Grub. Ubuntu 9.10 ships with Grub 2. Grub 2 works great with Xen, unless you use fakeraid. If you use fakeraid, I would suggest downgrading to Grub 1.

Ubuntu’s update-grub command won’t recognize your new Xen kernel, but you can easily modify the /etc/grub.d/40_custom script to manually insert it into your Grub configuration. Here’s what my 40_custom file looks like:

#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.

menuentry "Ubuntu, Xen 3.4.2 dom0 2.6.31.8-xen0-amd64" {
   set quiet=1
   insmod raid
   insmod mdraid
   insmod ext2
   set root=(md1)
   search --no-floppy --fs-uuid --set 31d4873b-fed1-49d9-b37d-89d9f7a46566
   multiboot (md1)/xen-3.4.2.gz vga=gfx-1440x900x32 quiet
   module (md1)/vmlinuz-2.6.31.8-xen0-amd64 root=/dev/mapper/vg0-root root=/dev/mapper/vg0-root ro quiet splash
   module (md1)/initrd.img-2.6.31.8-xen0-amd64
}

You will need to adjust this for your system. You can use the blkid command to get the UUID for your disk partitions. You might notice that I passed the root= parameter twice on the module line. This may be a bug somewhere in Xen or Grub but I could not get anything to work when only including this parameter once.

You also need to update your default operating system. You may want to wait until after you’ve verified everything works, but eventually you should edit /etc/default/grub. Mine looks like:

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.

GRUB_DEFAULT="Ubuntu, Xen 3.4.2 dom0 2.6.31.8-xen0-amd64"
#GRUB_DEFAULT="0"
#GRUB_HIDDEN_TIMEOUT=1
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT="3"
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX=""

# The resolution used on graphical terminal
GRUB_GFXMODE=1440x900x32

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entrys
#GRUB_DISABLE_LINUX_RECOVERY="true"

GRUB_DISABLE_OS_PROBER=true

Now run update-grub:

Reboot

Cross your fingers, it’s time to reboot into the Xen hypervisor and your new Xen kernel. If you are lucky everything will work and you’ll soon be back to your desktop. You may run into driver issues, in which case you’ll need to boot your original kernel and try your hand at reconfiguring and recompiling your kernel.

If you use Nvidia or ATI binary drivers they won’t work with the Xen kernel (yet). Enabling these is a subject for another day, but it can be done. For now, let’s stay focused on getting Xen fully installed and running.

After rebooting, you can check a couple of things:

Start Xen tools

Once you are running the Xen hypervisor and a Xen compatible dom0 kernel, you need to start the Xen daemon. This is done with:

You can start this automatically by establishing sysvinit links with:

If you want Xen domUs in /etc/xen/auto to start automatically on boot, set the xendomains init script to run on boot:

Compile a domU kernel

This step is optional. You may want to compile a kernel for use by your Linux domUs. Many distributions can run as a domU with their stock kernel however if that doesn’t work you can configure them to use a Xen domU kernel.

Configuring a Xen domU kernel is similar to configuring a Xen dom0 kernel:

When configuring the domU kernel, you’ll want to make sure you enable:

Processor type and features --->
  [*] Paravirtualized guest support --->
    [*] Xen guest support
Device drivers --->
  [*] Xen /dev/xen/evtchn device
  <*> Xen filesystem
  [*] Create xen entries under /sys/hypervisor

Ready for domUs

Congratulations, your Ubuntu 9.10 Karmic Koala installation is now running as a domain 0 on top of the latest Xen hypervisor. You are ready to start installing domUs.