Ubuntu

Now after building the paravirtualized Linux kernel for Dom0 requirements I will talk about what needs to be done in order to compile Xen hypervisor, including needed tools, and the configuration requirements for booting a priviledged guest (Ubuntu 9.04 Dom0 Linux image) on top of the ring0 Xen hyperviser software.

• Building Xen from sources

First we need to get the official release of xen sources from www.xen.org:

matmih@Express2:~/Work$ wget http://bits.xensource.com/oss-xen/release/3.4.1/xen-3.4.1.tar.gz
matmih@Express2:~/Work$ tar -xzvf /xen-3.4.1.tar.gz

Before compiling the hypervisor binaries and tools we must make sure that we have all the required dependency packages. An easy way to do this in Ubuntu, using apt, is to search for all the packages that start with xen, and then use build-dep apt option for those xen packages to install the dependencies:

matmih@Express2:~$ sudo apt-cache search xen
# now look for the packages that seem to be related to the Xen hypervisor and do the following:
matmih@Express2:~$ sudo apt-get build-dep xen-docs-3.3 xen-hypervisor-3.3 xen-shell xen-tools xen-utils-3.3
# a small list of required packages are listed below
matmih@Express2:~$ sudo apt-get install build-essential fakeroot crash kexec-tools makedumpfile git-core libncurses5 \
libncurses5-dev python python2.6-dev iproute bridge-utils libcurl-dev3 xserver-xorg-dev mercurial gitk uuid-dev gawk gettext texinfo bcc

Now we are ready to compile the Xen hypervisor, stubdoms and tools needed in Dom0 image:

# before compiling Xen make sure you set in xen3.4.1/Config.mk file:
# PYTHON = python
# PYTHON_PREFIX_ARG =
# Tuning Config.mk results Xen packages to be placed into usr/local/lib/python2.6/dist-packages due to Changeset 19594 in xen-3.4-testing.hg.
# Xen packages should go to usr/lib/python2.6/site-packages, which is not default location for python 2.6 on Ubuntu 9.04 ( vs F11 ),
# otherwise xend daemon will not start. You can always manually copy the python Xen libraries from the install location to the python's
# version correct directory if things still are not properly installed
matmih@Express2:~/Work/xen-3.4.1$ sudo make install-xen
matmih@Express2:~/Work/xen-3.4.1$ sudo make install-tool
matmih@Express2:~/Work/xen-3.4.1$ sudo make install-stubdom

The installed files and newly created directories in you local file system are also located in xen-3.4.1/dist. You can also use the dist/install.sh script to install dist directory content in your root filesystem.

• Deploying Xen

Now you should have everything installed, the binary kernel images including Xen should be in your /boot directory as below:

matmih@Express2:~$ ls /boot
abi-2.6.28-11-generic     config-2.6.31.5          initrd-2.6-xen.img            System.map-2.6.18.8-xen       vmcoreinfo-2.6.28-11-generic  vmlinuz-2.6.31.5  xen-3.gz
config-2.6.18.8-xen       grub                     initrd.img-2.6.28-11-generic  System.map-2.6.28-11-generic  vmlinuz-2.6.18.8-xen          xen-3.4.1.gz      xen.gz
config-2.6.28-11-generic  initrd-2.6.18.8-xen.img  memtest86+.bin                System.map-2.6.31.5           vmlinuz-2.6.28-11-generic     xen-3.4.gz        xen-syms-3.4.1

You can notice the hypervisor binaries with the corresponding symbolic links (xen-3.4.1.gz), the original 2.6.28-11 kernel that came with Ubuntu 9.04, the 2.6.18.8-xen version that was build using “make world” in Xen sources and finally the 2.6.31.5 corresponding to the paravirt_ops kernel version that I will use from now on. The hopefully last thing that you need to do before booting Xen is to add the correct entries to the Grub boot utility configuration file, so:

matmih@Express2:~$ sudo vim /boot/grub/menu.lst

# This boot entry contains the kernel image from Xen sources, ext3 root filesystem found on SATA drive partition /dev/sda5
title               Ubuntu 9.04, Xen Default
kernel              /boot/xen-3.4.1.gz
module              /boot/vmlinuz-2.6.18.8-xen root=/dev/sda5 ro console=tty0
module              /boot/initrd-2.6.18.8-xen.img
# This boot entry contains the paravirt_ops kernel compiled earlier, ext3 root filesystem found on drive uuid=505b3e6c-d4c1-4f48-9e4e-b54c2c46c41f=/dev/sda5
title               Ubuntu 9.04, Xen ParaVirtOps
uuid                505b3e6c-d4c1-4f48-9e4e-b54c2c46c41f
kernel              /boot/xen-3.4.1.gz
module              /boot/vmlinuz-2.6.31.5 root=UUID=505b3e6c-d4c1-4f48-9e4e-b54c2c46c41f ro console=tty0
module              /boot/initrd-2.6-xen.img

# This entry represents the original systems as found in the initial menu.lst file
title               Ubuntu 9.04, kernel 2.6.28-11-generic
uuid                505b3e6c-d4c1-4f48-9e4e-b54c2c46c41f
kernel              /boot/vmlinuz-2.6.28-11-generic root=UUID=505b3e6c-d4c1-4f48-9e4e-b54c2c46c41f ro quiet splash
initrd              /boot/initrd.img-2.6.28-11-generic

# This entry automatically added by the Debian installer for a non-linux OS on /dev/sda3
title Windows Vista (loader)
rootnoverify (hd0,2)
savedefault
makeactive
chainloader +1

You can now restart and try to boot one of the Xen configuration. For me only the second one worked with Ubuntu 9.04, the Default Xen configuration worked only if no mem_dom0 parameter was specified and had some boot errors (expecially regarding udev compatibility with /sys sysfs filesystem layout, and some memory window errors) which lead to the boot process taking a long time while mounting the file system and to the Xserver becaming unresponsive after some seconds/minutes in spent Dom0. I did not spent any more time investingating the problems because the paravirt_ops newer kernel worked like a charm. So basically you must always make sure that you use the same or at least a newer kernel version modified for a Xen Dom0 image that the one your system image came with. This will most likely reduce your deployment time by several days.

Another problem that I ran across was listing the current machines in my Xen Linux Dom0:

matmih@Express2:~$ sudo xm list
Error: Unable to connect to xend: No such file or directory. Is xend running?

Aparently this was caused by the xend daemon not even been running, you can easialy check this.  When I tried to manually start it I got the following:

matmih@Express2:~$ ps ax | grep xend # to check whether xend is running
matmih@Express2:~$ sudo /usr/sbin/xend start # xend should start automatically, check the xend script in /etc/init.d directory
ERROR Internal error: Could not obtain handle on privileged command interface (2 = No such file or directory)

Depending on the Xen/xend version you are running the python script will check either the sysfs exposed /sys/hypervisor directory or the xenfs‘s /proc/xen. In the current unstable branch of xen-tools, the file libxc/xc_linux.c looks for /proc/xen/privcmd.
But this may or may not work. First, the latest tip for the Dom0 kernel makes the whole /proc/xen entity an option. Presumably, it will be superseded by /sys/hypervisor, which itself is optional. So make sure that you booted Xen Linux and that you have the above directories for the coresponding filesystems mounted. If you do not see any contents in /sys/hypervisor directory than you may need to get an erlier version  of Xen, or Xen support has not been added to your Dom0 kernel. If there is nothing in /proc/xen you may need to mount xenfs filesystem on that location:

matmih@Express2:~$ cat /etc/fstab
# <file system>                            <mount point>   <type>        <options>                    <dump>  <pass>
proc                                        /proc           proc          defaults                     0       0
# / was on /dev/sda5 during installation
UUID=505b3e6c-d4c1-4f48-9e4e-b54c2c46c41f   /               ext3          relatime,errors=remount-ro   0       1
/dev/scd0                                   /media/cdrom0   udf,iso9660   user,noauto,exec,utf8        0       0
none                                        /proc/xen       xenfs         defaults                     0       0

A current view of my system’s mounts and the content of the hypervisor new filesystem entries can be found below:

matmih@Express2:~$ ls /sys/hypervisor/
compilation  properties  type  uuid  version
matmih@Express2:~$ ls /proc/xen/
capabilities  privcmd  xenbus  xsd_kva  xsd_port
matmih@Express2:~$  df -ah
Filesystem            Size  Used Avail Use% Mounted on/l
tmpfs                 1.5G     0  1.5G   0% /lib/init/rw
proc                     0     0     0   -  /proc
sysfs                    0     0     0   -  /sys
varrun                1.5G  108K  1.5G   1% /var/run
varlock               1.5G  4.0K  1.5G   1% /var/lock
udev                  1.5G  184K  1.5G   1% /dev
tmpfs                 1.5G   76K  1.5G   1% /dev/shm
devpts                   0     0     0   -  /dev/pts
fusectl                  0     0     0   -  /sys/fs/fuse/connections
none                     0     0     0   -  /proc/xen
securityfs               0     0     0   -  /sys/kernel/security
binfmt_misc              0     0     0   -  /proc/sys/fs/binfmt_misc
gvfs-fuse-daemon         0     0     0   -  /home/matmih/.gvfs
/dev/sr1               37M   37M     0 100% /media/VMC LITE 9.3.0.9237

The only problem that I can see is that even if my 1920×1080 resolution was kept (I had to install ati drivers to get 16:9 aspect ration working on Ubuntu 9.04) the refresh rate seems to be low as if I am running a Vesa driver for my video card. The resons seems to be that the Ati Radeon driver, flgrx,  has not been installed for my current kernel (lsmod | grep flgrx). The ati-driver-installer-9-9-x86.x86_64.run installer, appart from configuring my Xserver, has installed some drivers for my previous kernel (2.6.28-11). I need to rebuild and install the kernel drivers for my current paravirt_ops Xen 2.6.31-5 kernel, but running the installer again did not succeed to build the flgrx module again due to the following error:

matmih@Express2:~$ cat /usr/share/ati/fglrx-install.log
......
Building modules, stage 2.
 MODPOST 1 modules
 WARNING: "find_task_by_vpid" [/lib/modules/fglrx/build_mod/2.6.x/fglrx.ko] undefined!
.....

It seems that the find_task_by_vpid function has been removed starting with 2.6.30 kernel version. To fix this go to the fldrx driver sources location and do the following:

matmih@Express2:~$ cd /lib/modules/fglrx/build_mod
matmih@Express2:~:/lib/modules/fglrx/build_mod$ sudo wget http://launchpadlibrarian.net/29698859/find_task_by_vpid.patch
matmih@Express2:~:/lib/modules/fglrx/build_mod$ sudo patch -p1 < find_task_by_vpid.patch
matmih@Express2:~:/lib/modules/fglrx/build_mod$ sudo ./make.sh
matmih@Express2:~:/lib/modules/fglrx/build_mod$ cd ..
matmih@Express2:~:/lib/modules/fglrx$ sudo ./make_install.sh

Unfortunatly this did not fix my initial problems. Running dmesg command I could see that the flgrx module could not be loaded due to the following error message: “[fglrx] CPU does not support PAT”.

There are several ways of building a paravirtualized Linux kernel that can be used for a Dom0 or DomU.

• Default kernel from xen package

In xen3.4.1 sources the modified kernel is based on a 2.6.18.8 version. In order to build everything, including downloading the kernel sources you will need to go to xen directory and:

matmih@Express2:~/Work/xen-3.4.1$ make world

Note*: you will need mercurial packages in order to download the kernel.

You can use the default settings for the kernel configuration (including PCI passthrough in order to keep the same virtual <-> physical mappings of the PCI devices between Dom0 and guest partitions). The built kernel image will be placed at dist/install/boot/vmlinuz-2.6.18.8-xen. Also you will need to configure a ramdisk for the kernel to boot with your Linux distribution. If you need to rebuild the kernel of modify some configs you can use:

make linux-2.6-xen-config CONFIGMODE=menuconfig     (or xconfig)
make linux-2.6-xen-build
make linux-2.6-xen-install

This should be the easiest and safest mode to compile a paravirtualized kernel for Xen hypervisor, and should work for most distributions and filesystems images. But for my Ubuntu 9.04 this kernel is a bit old due to udev version and it was quite unstable, the boot log also had some problems and could not boot without assigning the whole RAM memory (specifying the dom0_mem parameter in grub script cause it to crash during boot time with “Out of Memory: Kill process 2 (migration/0) score 0 and children”) and even if I assigned all the physical memory it booted (though it took a very long time) but after a few seconds in gnome desktop it did not respond any more (no input was possible only hard reset).

So as a personal rule I should always use if not the same at least a newer kernel version than the one that came with my Ubuntu 9.04 distribution.

• Vanilla kernel + Opensuse Xen patches

First we need to get the kernel sources for the version we want to patch using Andrew Lyon’s mentained patches for a Xen Dom0 kernel. A  good thing to do is to check out Gentoo Xen Patches to see the latest Xen patches version and the vanila kernel we want to download from the Linux Kernel Archive. From Andrew Lyon’s download page we we’ll get the latest xen patches, 2.6.31-7 based on Opensuse’s kernel sources 2.6.31-5. So we do the following:

matmih@Express2:~/Work$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.31.5.tar.bz2
matmih@Express2:~/Work$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.31-7.tar.bz2
matmih@Express2:~/Work$ bzcat linux-2.6.31.tar.bz2 | tar xf -
matmih@Express2:~/Work$ cd linux-2.6.31 ; mkdir xen-patches ; cd xen-patches
matmih@Express2:~/Work/linux-2.6.31/xen-patches$ bzcat ../../xen-patches-2.6.31-7.tar.bz2 | tar xf - ; cd ..
# now apply the pathes ... should be alphabetically ordered in the archive and will only print errors
matmih@Express2:~/Work/linux-2.6.31$ cat xen-patches/6*.patch* | patch -p1 -s -l

You now have a possible correct kernel environment that you can configure, build and install for a Dom0 image.

• Xen paravirt_ops Linux kernel

A new kernel infrastructure that allows it to run paravirtualized on a hypervisor, like Xen, VMWare’s VMI and lguest. The infrastructure, supporting x86_32, x86_64 and ia64 architectures, allows you to compile a single kernel binary which will either boot native on bare hardware (or in hvm mode under Xen), or boot fully paravirtualized in any of the environments you’ve enabled in the kernel configuration. Xen pv_ops (domU) support has been in mainline Linux since 2.6.23, and is the basis of all on-going Linux/Xen development. In xen 3.5 it will be the default paravirtualized Dom0 kernel that comes with the hypervizor package. Although we are using the stable version of xen 3.4 we are going to use this kernel as it may as well be the most common way of building a Dom0 kernel in the future. You can find more information at XenParavirtOps page. First we need to get the latest sourcing unsing git found in git-core package:

matmih@Express2:~/Work$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen$
matmih@Express2:~/Work$ cd linux-2.6-xen
matmih@Express2:~/Work/linux-2.6-xen$ git pull

Now that we have a valid source kernel environment, made with either methods, though I recommend the last one, we should begin configuring the kernel to be build for your image and hardware requirements, as well as for the requirements of a paravirtualized kernel that can run as a Dom0 kernel, or even in a DomU guest partition. Normally you should copy the kernel configuration that came with your distribution (found in /boot/config-2.6.28- 11-generic for my 64bit Ubuntu 9.04) to linux-2.6-xen/.config and start from there addind Xen features as you find them, as well as the devices may not be included, either in the binary or as modules (use lspci command to see which drivers you need on your machine). If you are using the paravirt-ops kernel, as I did the only thing that you must enable is Dom0 support, so:

matmih@Express2:~/Work/linux-2.6-xen$ make menuconfig
# and enable the following
# Processor type and features --->
#    [*] Paravirtualized guest support --->
#           [*] Enable Xen priviledged domain support (NEW)

And the build and install the kernel image. Do not forget to also build the initial ramdisk that will contain some of the modules required for Dom0 to boot.

matmih@Express2:~/Work/linux-2.6-xen$ make
matmih@Express2:~/Work/linux-2.6-xen$ sudo make modules_install install
matmih@Express2:~/Work/linux-2.6-xen$ ls /lib/modules ; ls /boot # to see what was the name of the kernel newly installed
matmih@Express2:~/Work/linux-2.6-xen$ sudo mkinitramfs -o /boot/initrd-2.6-xen.img 2.6.31.5

You should now have the kernel image and initial ramdisk located in your local /boot directory. Also, expecially if you have a SATA disk drive, do not forget to force adding the following modules to your image, if not already built into the kernel image: aacraid, sd_mod, scsi_mod. A detailed description about the SCSI system, for 2.4 kernel at least, can be found at here, and an architectural overview of the implementation can be seen bellow:

So basically you must do the following things to make sure that the above modules are in your ramdisk, depending if you are using mkinitrd, or mkinitramfs utility, where 2.6.31.5 is the kernel version you want to create the image for:

matmih@Express2:~$ sudo depmode 2.6.31.5
matmih@Express2:~$ sudo mkinitrd -v -f --with=aacraid --with=sd_mod --with=scsi_mod /boot/initrd-xen.img 2.6.31.5
# or add the modules to mkinitramfs utility configuration file like I did and run the utility command
matmih@Express2:~$ cat /etc/initramfs-tools/modules
aacraid
sd_mod
scsi_mod
matmih@Express2:~$ sudo mkinitramfs -o /boot/initrd-2.6-xen.img 2.6.31.5

Usually a common error message when the kernel tries to boot is usually “VFS: Cannot open root device “sda1″ or unknown-block(0,0)“. Also if the boot log indicates that it cannot mount the filesystem you should check if you also added (or built in the kernel image) the required filesystem (like ext3). You can force adding this module as well like above I did for the SCSI SATA disk requirements, but running depmod for the installed modules of the built kernel should add the above modules (you can see what depmod generated looking at, for example, /lib/modules/2.6.31.5/modules.dep file used by the initial ramdisk creation utilities to add the modules to the compressed initrd image).

matmih@Express2:~/Work/xen-3.4.1$ make world

make linux-2.6-xen-config CONFIGMODE=menuconfig     (or xconfig)
make linux-2.6-xen-build
make linux-2.6-xen-install

matmih@Express2:~/Work$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.31.5.tar.bz2
matmih@Express2:~/Work$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.31-7.tar.bz2
matmih@Express2:~/Work$ bzcat linux-2.6.31.tar.bz2 | tar xf -
matmih@Express2:~/Work$ cd linux-2.6.31 ; mkdir xen-patches ; cd xen-patches
matmih@Express2:~/Work/linux-2.6.31/xen-patches$ bzcat ../../xen-patches-2.6.31-7.tar.bz2 | tar xf - ; cd ..
# now apply the pathes ... should be alphabetically ordered in the archive and will only print errors
matmih@Express2:~/Work/linux-2.6.31$ cat xen-patches/6*.patch* | patch -p1 -s -l

matmih@Express2:~/Work$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen$
matmih@Express2:~/Work$ cd linux-2.6-xen
matmih@Express2:~/Work/linux-2.6-xen$ git pull

matmih@Express2:~/Work/linux-2.6-xen$ make menuconfig
# and enable the following
# Processor type and features --->
#    [*] Paravirtualized guest support --->
#           [*] Enable Xen priviledged domain support (NEW)

matmih@Express2:~/Work/linux-2.6-xen$ make
matmih@Express2:~/Work/linux-2.6-xen$ sudo make modules_install install
matmih@Express2:~/Work/linux-2.6-xen$ ls /lib/modules ; ls /boot # to see what was the name of the kernel newly installed
matmih@Express2:~/Work/linux-2.6-xen$ sudo mkinitramfs -o /boot/initrd-2.6-xen.img 2.6.31.5