Now after building the paravirtualized Linux kernel for Dom0 requirements I will talk about what needs to be done in order to compile Xen hypervisor, including needed tools, and the configuration requirements for booting a priviledged guest (Ubuntu 9.04 Dom0 Linux image) on top of the ring0 Xen hyperviser software.
- Building Xen from sources
First we need to get the official release of xen sources from www.xen.org:
matmih@Express2:~/Work$ wget http://bits.xensource.com/oss-xen/release/3.4.1/xen-3.4.1.tar.gz
matmih@Express2:~/Work$ tar -xzvf /xen-3.4.1.tar.gz
Before compiling the hypervisor binaries and tools we must make sure that we have all the required dependency packages. An easy way to do this in Ubuntu, using apt, is to search for all the packages that start with xen, and then use build-dep apt option for those xen packages to install the dependencies:
matmih@Express2:~$ sudo apt-cache search xen
# now look for the packages that seem to be related to the Xen hypervisor and do the following:
matmih@Express2:~$ sudo apt-get build-dep xen-docs-3.3 xen-hypervisor-3.3 xen-shell xen-tools xen-utils-3.3
# a small list of required packages are listed below
matmih@Express2:~$ sudo apt-get install build-essential fakeroot crash kexec-tools makedumpfile git-core libncurses5 \
libncurses5-dev python python2.6-dev iproute bridge-utils libcurl-dev3 xserver-xorg-dev mercurial gitk uuid-dev gawk gettext texinfo bcc
Now we are ready to compile the Xen hypervisor, stubdoms and tools needed in Dom0 image:
# before compiling Xen make sure you set in xen3.4.1/Config.mk file:#PYTHON = python#PYTHON_PREFIX_ARG =# Tuning Config.mk results Xen packages to be placed into usr/local/lib/python2.6/dist-packages due to Changeset 19594 in xen-3.4-testing.hg. # Xen packages should go to usr/lib/python2.6/site-packages, which is not default location for python 2.6 on Ubuntu 9.04 ( vs F11 ), # otherwise xend daemon will not start. You can always manually copy the python Xen libraries from the install location to the python's # version correct directory if things still are not properly installed matmih@Express2:~/Work/xen-3.4.1$ sudo make install-xen matmih@Express2:~/Work/xen-3.4.1$ sudo make install-tool matmih@Express2:~/Work/xen-3.4.1$ sudo make install-stubdom
The installed files and newly created directories in you local file system are also located in xen-3.4.1/dist. You can also use the dist/install.sh script to install dist directory content in your root filesystem.
- Deploying Xen
Now you should have everything installed, the binary kernel images including Xen should be in your /boot directory as below:
matmih@Express2:~$ ls /boot abi-2.6.28-11-generic config-2.6.31.5 initrd-2.6-xen.img System.map-2.6.18.8-xen vmcoreinfo-2.6.28-11-generic vmlinuz-2.6.31.5 xen-3.gz config-2.6.18.8-xen grub initrd.img-2.6.28-11-generic System.map-2.6.28-11-generic vmlinuz-2.6.18.8-xen xen-3.4.1.gz xen.gz config-2.6.28-11-generic initrd-2.6.18.8-xen.img memtest86+.bin System.map-2.6.31.5 vmlinuz-2.6.28-11-generic xen-3.4.gz xen-syms-3.4.1
You can notice the hypervisor binaries with the corresponding symbolic links (xen-3.4.1.gz), the original 2.6.28-11 kernel that came with Ubuntu 9.04, the 2.6.18.8-xen version that was build using “make world” in Xen sources and finally the 2.6.31.5 corresponding to the paravirt_ops kernel version that I will use from now on. The hopefully last thing that you need to do before booting Xen is to add the correct entries to the Grub boot utility configuration file, so:
matmih@Express2:~$ sudo vim /boot/grub/menu.lst
# This boot entry contains the kernel image from Xen sources, ext3 root filesystem found on SATA drive partition /dev/sda5 title Ubuntu 9.04, Xen Default kernel /boot/xen-3.4.1.gz module /boot/vmlinuz-2.6.18.8-xen root=/dev/sda5 ro console=tty0 module /boot/initrd-2.6.18.8-xen.img # This boot entry contains the paravirt_ops kernel compiled earlier, ext3 root filesystem found on drive uuid=505b3e6c-d4c1-4f48-9e4e-b54c2c46c41f=/dev/sda5 title Ubuntu 9.04, Xen ParaVirtOps uuid 505b3e6c-d4c1-4f48-9e4e-b54c2c46c41f kernel /boot/xen-3.4.1.gz module /boot/vmlinuz-2.6.31.5 root=UUID=505b3e6c-d4c1-4f48-9e4e-b54c2c46c41f ro console=tty0 module /boot/initrd-2.6-xen.img
# This entry represents the original systems as found in the initial menu.lst file title Ubuntu 9.04, kernel 2.6.28-11-generic uuid 505b3e6c-d4c1-4f48-9e4e-b54c2c46c41f kernel /boot/vmlinuz-2.6.28-11-generic root=UUID=505b3e6c-d4c1-4f48-9e4e-b54c2c46c41f ro quiet splash initrd /boot/initrd.img-2.6.28-11-generic # This entry automatically added by the Debian installer for a non-linux OS on /dev/sda3 title Windows Vista (loader) rootnoverify (hd0,2) savedefault makeactive chainloader +1
You can now restart and try to boot one of the Xen configuration. For me only the second one worked with Ubuntu 9.04, the Default Xen configuration worked only if no mem_dom0 parameter was specified and had some boot errors (expecially regarding udev compatibility with /sys sysfs filesystem layout, and some memory window errors) which lead to the boot process taking a long time while mounting the file system and to the Xserver becaming unresponsive after some seconds/minutes in spent Dom0. I did not spent any more time investingating the problems because the paravirt_ops newer kernel worked like a charm. So basically you must always make sure that you use the same or at least a newer kernel version modified for a Xen Dom0 image that the one your system image came with. This will most likely reduce your deployment time by several days.
Another problem that I ran across was listing the current machines in my Xen Linux Dom0:
matmih@Express2:~$ sudo xm list
Error: Unable to connect to xend: No such file or directory. Is xend running?
Aparently this was caused by the xend daemon not even been running, you can easialy check this. When I tried to manually start it I got the following:
matmih@Express2:~$ ps ax | grep xend # to check whether xend is running matmih@Express2:~$ sudo /usr/sbin/xend start # xend should start automatically, check the xend script in /etc/init.d directory ERROR Internal error: Could not obtain handle on privileged command interface (2 = No such file or directory)
Depending on the Xen/xend version you are running the python script will check either the sysfs exposed /sys/hypervisor directory or the xenfs‘s /proc/xen. In the current unstable branch of xen-tools, the file libxc/xc_linux.c looks for /proc/xen/privcmd.
But this may or may not work. First, the latest tip for the Dom0 kernel makes the whole /proc/xen entity an option. Presumably, it will be superseded by /sys/hypervisor, which itself is optional. So make sure that you booted Xen Linux and that you have the above directories for the coresponding filesystems mounted. If you do not see any contents in /sys/hypervisor directory than you may need to get an erlier version of Xen, or Xen support has not been added to your Dom0 kernel. If there is nothing in /proc/xen you may need to mount xenfs filesystem on that location:
matmih@Express2:~$ cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
# / was on /dev/sda5 during installation
UUID=505b3e6c-d4c1-4f48-9e4e-b54c2c46c41f / ext3 relatime,errors=remount-ro 0 1
/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto,exec,utf8 0 0
none /proc/xen xenfs defaults 0 0
A current view of my system’s mounts and the content of the hypervisor new filesystem entries can be found below:
matmih@Express2:~$ ls /sys/hypervisor/ compilation properties type uuid version matmih@Express2:~$ ls /proc/xen/ capabilities privcmd xenbus xsd_kva xsd_port matmih@Express2:~$ df -ah Filesystem Size Used Avail Use% Mounted on/l tmpfs 1.5G 0 1.5G 0% /lib/init/rw proc 0 0 0 - /proc sysfs 0 0 0 - /sys varrun 1.5G 108K 1.5G 1% /var/run varlock 1.5G 4.0K 1.5G 1% /var/lock udev 1.5G 184K 1.5G 1% /dev tmpfs 1.5G 76K 1.5G 1% /dev/shm devpts 0 0 0 - /dev/pts fusectl 0 0 0 - /sys/fs/fuse/connections none 0 0 0 - /proc/xen securityfs 0 0 0 - /sys/kernel/security binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc gvfs-fuse-daemon 0 0 0 - /home/matmih/.gvfs /dev/sr1 37M 37M 0 100% /media/VMC LITE 9.3.0.9237
The only problem that I can see is that even if my 1920×1080 resolution was kept (I had to install ati drivers to get 16:9 aspect ration working on Ubuntu 9.04) the refresh rate seems to be low as if I am running a Vesa driver for my video card. The resons seems to be that the Ati Radeon driver, flgrx, has not been installed for my current kernel (lsmod | grep flgrx). The ati-driver-installer-9-9-x86.x86_64.run installer, appart from configuring my Xserver, has installed some drivers for my previous kernel (2.6.28-11). I need to rebuild and install the kernel drivers for my current paravirt_ops Xen 2.6.31-5 kernel, but running the installer again did not succeed to build the flgrx module again due to the following error:
matmih@Express2:~$ cat /usr/share/ati/fglrx-install.log
......
Building modules, stage 2.
MODPOST 1 modules
WARNING: "find_task_by_vpid" [/lib/modules/fglrx/build_mod/2.6.x/fglrx.ko] undefined!
.....
It seems that the find_task_by_vpid function has been removed starting with 2.6.30 kernel version. To fix this go to the fldrx driver sources location and do the following:
matmih@Express2:~$ cd /lib/modules/fglrx/build_mod matmih@Express2:~:/lib/modules/fglrx/build_mod$ sudo wget http://launchpadlibrarian.net/29698859/find_task_by_vpid.patch matmih@Express2:~:/lib/modules/fglrx/build_mod$ sudo patch -p1 < find_task_by_vpid.patch matmih@Express2:~:/lib/modules/fglrx/build_mod$ sudo ./make.sh matmih@Express2:~:/lib/modules/fglrx/build_mod$ cd .. matmih@Express2:~:/lib/modules/fglrx$ sudo ./make_install.sh
Unfortunatly this did not fix my initial problems. Running dmesg command I could see that the flgrx module could not be loaded due to the following error message: “[fglrx] CPU does not support PAT”.
Note, following this recipe and the building of a dom0 kernel recipe on a fresh karmic install, had issues with xend starting. Appears several of the frontend/backend xen drivers get built as modules if your don’t pay attention to the kernel config, once loaded xend starts.