Home > Server Admin > Installing Xen on Ubuntu 9.10

Installing Xen on Ubuntu 9.10

January 16th, 2010

Recently I switched to Ubuntu as the primary operating system on my desktop machine. As part of the switch, I wanted to install Xen to virtualize some additional operating systems. Xen provides very good performance when virtualizing Linux distributions due to paravirtualization. The latest versions can also virtualize certain unmodified guest operating systems on processors that support virtualization.

In a Xen setup, the Xen hypervisor runs directly on the hardware (bare-metal). The first guest operating system (dom0) runs “on top” of Xen and has full access to the underlying hardware. Additional guests (domU) also run on top of the Xen hypervisor, but with limited access to the underlying hardware.

Converting an existing Ubuntu install to a Xen dom0 install requires installing the Xen hypervisor. In previous versions of Ubuntu this could easily be done using apt. Unfortunately no packages exist for Ubuntu 9.10 (Karmic Koala). In this post I’ll describe the steps I took to install Xen from source on Karmic. If you’d rather install via binary packages you’ll need to find a third-party repository.

Install Xen hypervisor and tools

# In order to compile Xen, you must first install some build dependencies:
$ sudo apt-get install bin86 bcc iasl uuid-dev libsdl-dev

$ cd /usr/src

# See http://www.xen.org/products/xen_source.html for latest downloads
$ wget http://bits.xensource.com/oss-xen/release/3.4.2/xen-3.4.2.tar.gz
$ tar -xzf xen-3.4.2.tar.gz

# I created this small patch to adapt Xen to Ubuntu’s layout
$ wget http://www.bltweb.net/patches/xen-3.4.2-ubuntu.patch
# or if you are using Xen 4.0:
$ wget http://www.bltweb.net/patches/xen-4.0.0-ubuntu.patch

$ cd xen-3.4.2
$ patch -p1 < ../xen-3.4.2-ubuntu.patch
$ make dist-xen dist-tools dist-stubdom
$ sudo make install-xen install-tools install-stubdom

Install Xen Kernel Sources

The stock Karmic kernel will not work when running as dom0 in Xen. You must install a new kernel. I chose to compile my own kernel. Compiling a kernel is not for the faint of heart, but isn’t as hard as you might think.

You have two main choices when it comes to a dom0 kernel. The Xen kernel or a dom0 pv-ops kernel. Normally I would recommend using a pv-ops kernel. The same pv-ops kernel can run on bare-metal or under the Xen hypervisor. The pv-ops kernel is going to be the default in future versions of Xen and will most likely be included in the mainline Linux kernel soon.

Unfortunately, the pv-ops kernel will not work with binary graphics drivers provided by Nvidia. Since I have an Nvidia graphics card (and want to use the binary drivers) I need to use the standard Xen kernel. The standard Xen kernel is still version 2.6.18, however luckily Andrew Lyon maintains forward ported patches for Gentoo that we can use on our Ubuntu install.

First things first, let’s get the kernel sources downloaded and patched:

$ cd /usr/src

# Download the vanilla kernel sources from http://www.kernel.org/
$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.31.8.tar.bz2
$ tar -xjf linux-2.6.31.8.tar.bz2
$ mv linux-2.6.31.8 linux-2.6.31.8-xen0-amd64

# Download the forward ported xen patches from http://code.google.com/p/gentoo-xen-kernel/downloads/list
$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.31-10.tar.bz2
$ mkdir xenpatches-v10_2.6.31.8
$ cd xenpatches-v10_2.6.31.8
$ tar -xjf ../xen-patches-2.6.31-10.tar.bz2
$ cd ../

# Apply the xen patches
$ cd linux-2.6.31.8-xen0-amd64
$ for i in `ls ../xenpatches-v10_2.6.31.8/`; do patch -p1 < ../xenpatches-v10_2.6.31.8/"$i"; done

At this point I like to modify the kernel’s “extra version” to clearly identify that this is a dom0 kernel. To do this, modify the Makefile using your favorite text editor and change the EXTRAVERSION line to look like:

VERSION = 2
PATCHLEVEL = 6
SUBLEVEL = 31
EXTRAVERSION = .8-xen0-amd64
NAME = Man-Eating Seals of Antiquity
...

Compile Xen Kernel

Configuring your kernel is left as an exercise for the reader. There is a ton of existing documentation on how to do this. The easiest way to get started is to copy the configuration from the stock Ubuntu kernel and run make menuconfig.

$ cp /boot/config-2.6-31-17-generic .config
$ make menuconfig

There are some options that you want to be sure you enable:

Processor type and features --->
  [*] Enable Xen compatible kernel
Networking support --->
  Networking options --->
    <*> 802.1d Ethernet Bridging
Device Drivers --->
  XEN --->
    [*] Privileged Guest (domain 0)
    <M> Backend driver support
    <M>   Block-device backend driver
    <M>   Block-device tap backend driver
    <M>   Block-device tap backend driver 2
    <M>   Network-device backend driver
    <M>   PCI-device backend driver
    <M>   SCSI backend driver
    <M>   USB backend driver
    <*> Block-device frontend driver
    <*> Network-device frontend driver
    <M> SCSI frontend driver
    <M> USB frontend driver
    <*> Framebuffer-device frontend driver
    <*>   Keyboard-device frontend driver
    [*] Disable serial port drivers
    <*> Export Xen attributes in sysfs
        Xen version compatibility (3.3.0 and later) --->
           3.3.0 and later
 

Now compile and install your kernel:

# To speed up the compile, set CONCURRENCY_LEVEL to one higher than the number of processor cores in your machine
$ export CONCURRENCY_LEVEL=5
$ make
$ sudo make install
$ sudo make modules_install

# Make an initramfs
$ sudo mkinitramfs -o /boot/initrd.img-2.6.31.8-xen0-amd64 2.6.31.8-xen0-amd64

If you compiled the Xen net, block and usb backend drivers as modules, you will probably want to load them by default by adding them to your /etc/modules file:

# /etc/modules
...
# These modules are required for Xen DomUs to work
netbk
blkbk
usbbk

Update Grub

Most existing Xen documentation on the web refers to the first version of Grub. Ubuntu 9.10 ships with Grub 2. Grub 2 works great with Xen, unless you use fakeraid. If you use fakeraid, I would suggest downgrading to Grub 1.

Ubuntu’s update-grub command won’t recognize your new Xen kernel, but you can easily modify the /etc/grub.d/40_custom script to manually insert it into your Grub configuration. Here’s what my 40_custom file looks like:

#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.

menuentry "Ubuntu, Xen 3.4.2 dom0 2.6.31.8-xen0-amd64" {
   set quiet=1
   insmod raid
   insmod mdraid
   insmod ext2
   set root=(md1)
   search --no-floppy --fs-uuid --set 31d4873b-fed1-49d9-b37d-89d9f7a46566
   multiboot (md1)/xen-3.4.2.gz vga=gfx-1440x900x32 quiet
   module (md1)/vmlinuz-2.6.31.8-xen0-amd64 root=/dev/mapper/vg0-root root=/dev/mapper/vg0-root ro quiet splash
   module (md1)/initrd.img-2.6.31.8-xen0-amd64
}

You will need to adjust this for your system. You can use the blkid command to get the UUID for your disk partitions. You might notice that I passed the root= parameter twice on the module line. This may be a bug somewhere in Xen or Grub but I could not get anything to work when only including this parameter once.

You also need to update your default operating system. You may want to wait until after you’ve verified everything works, but eventually you should edit /etc/default/grub. Mine looks like:

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.

GRUB_DEFAULT="Ubuntu, Xen 3.4.2 dom0 2.6.31.8-xen0-amd64"
#GRUB_DEFAULT="0"
#GRUB_HIDDEN_TIMEOUT=1
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT="3"
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX=""

# The resolution used on graphical terminal
GRUB_GFXMODE=1440x900x32

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entrys
#GRUB_DISABLE_LINUX_RECOVERY="true"

GRUB_DISABLE_OS_PROBER=true

Now run update-grub:

$ sudo update-grub

Reboot

Cross your fingers, it’s time to reboot into the Xen hypervisor and your new Xen kernel. If you are lucky everything will work and you’ll soon be back to your desktop. You may run into driver issues, in which case you’ll need to boot your original kernel and try your hand at reconfiguring and recompiling your kernel.

If you use Nvidia or ATI binary drivers they won’t work with the Xen kernel (yet). Enabling these is a subject for another day, but it can be done. For now, let’s stay focused on getting Xen fully installed and running.

After rebooting, you can check a couple of things:

$ uname -r
2.6.31.8-xen0-amd64
$ sudo cat /proc/xen/capabilities
control_d

Start Xen tools

Once you are running the Xen hypervisor and a Xen compatible dom0 kernel, you need to start the Xen daemon. This is done with:

$ sudo /etc/init.d/xend start

You can start this automatically by establishing sysvinit links with:

$ sudo update-rc.d xend start 20 2 3 4 5 . stop 80 0 1 6 .

If you want Xen domUs in /etc/xen/auto to start automatically on boot, set the xendomains init script to run on boot:

$ sudo update-rc.d xendomains start 21 2 3 4 5 . stop 79 0 1 6 .

Compile a domU kernel

This step is optional. You may want to compile a kernel for use by your Linux domUs. Many distributions can run as a domU with their stock kernel however if that doesn’t work you can configure them to use a Xen domU kernel.

Configuring a Xen domU kernel is similar to configuring a Xen dom0 kernel:

$ cd /usr/src

$ tar -xjf linux-2.6.31.8.tar.bz2
$ mv linux-2.6.31.8 linux-2.6.31.8-xenU-amd64

# Modify Makefile so EXTRAVERSION = .8-xenU-amd64

$ cp /boot/config-2.6-31-17-generic .config
$ make menuconfig
$ sudo make install
$ sudo make modules_install

When configuring the domU kernel, you’ll want to make sure you enable:

Processor type and features --->
  [*] Paravirtualized guest support --->
    [*] Xen guest support
Device drivers --->
  [*] Xen /dev/xen/evtchn device
  <*> Xen filesystem
  [*] Create xen entries under /sys/hypervisor  

Ready for domUs

Congratulations, your Ubuntu 9.10 Karmic Koala installation is now running as a domain 0 on top of the latest Xen hypervisor. You are ready to start installing domUs.

Server Admin , ,

  1. October 7th, 2015 at 08:46 | #1

    hmmm, how did you perform the inaltsl in reference to intsalling grub,, and replacing ntldr?, are you saying it is loading the windows media screen and then failing?, or grub is loading and failing? if the first u may have corrupted system files on the windows system partition possibly checkdsk or fixboot will repair,, if the second u should probably boot into ubuntu and fix the grub menu to properly point at the windows active sys partition =) becareful

  2. October 11th, 2015 at 04:42 | #2

    ο ε ο ε ο ε ο εο ε ο background ου ? ο ε

  3. December 10th, 2015 at 02:25 | #3

    Hey hey hey, take a gander at what’ you’ve done

  4. December 10th, 2015 at 07:04 | #4

    Thinking like that shows an expert’s touch

  5. February 6th, 2016 at 16:19 | #5

    Od jutra rozpoczynam odchudzanie, kto się odchudza ze mna?
    Znalazlam w google dobry sposob na chudnięcie, wygoglujcie sobie – xxally mój sukces
    w odchudzaniu

  6. June 5th, 2016 at 09:33 | #6

    Oot ihana Hanna, oikea hyvän olon lähde! Sain tänään palautettua 9 kk ahertamani diplomityöni, huhhuh mikä urakka, onhan se kuitenkin itse kirjoittamani "kirja". Osaat varmaan samaistua tunteisiini ;) On yhtäaikaa tyhjä, iloinen, helpottunut, ja niin kovin outo…olo. Nyt on pitkästä aikaa mahdollisuus antaa aikaa blogeillekin, tulin tietty hakemaan täältä iloista mieltä! Saat jotenkin vangittua elämän kauneuden kuviisi! -Marissa

  7. June 7th, 2016 at 06:58 | #7

    Thank You for this clear & honest sharing….. it seems we have a lot in common, including your ankle and me my left foot this June. All experiences, stepping stones on our journey to heal.Good Luck and Blessings to you. Keep well, Brigitte

  8. August 20th, 2016 at 12:04 | #8

    "The statistics speak for themselves, and so has its chilling effect on America's status as the worlds #1 jailer"…NOT thinking clearly are you?People in this country get the government they deserve by what they do (or don't do) at the polling booth…

Comment pages
  1. January 16th, 2010 at 16:22 | #1
  2. January 16th, 2010 at 21:29 | #2
  3. January 17th, 2010 at 13:57 | #3
  4. January 18th, 2010 at 22:11 | #4
  5. January 19th, 2010 at 20:53 | #5
  6. March 7th, 2010 at 07:37 | #6
  7. March 18th, 2010 at 13:32 | #7
  8. April 13th, 2011 at 08:48 | #8