2009 November 2
Building a Linux kernel for Xen’s Dom0 image
There are several ways of building a paravirtualized Linux kernel that can be used for a Dom0 or DomU.
- Default kernel from xen package
In xen3.4.1 sources the modified kernel is based on a 2.6.18.8 version. In order to build everything, including downloading the kernel sources you will need to go to xen directory and:
matmih@Express2:~/Work/xen-3.4.1$ make world
Note*: you will need mercurial packages in order to download the kernel.
You can use the default settings for the kernel configuration (including PCI passthrough in order to keep the same virtual <-> physical mappings of the PCI devices between Dom0 and guest partitions). The built kernel image will be placed at dist/install/boot/vmlinuz-2.6.18.8-xen. Also you will need to configure a ramdisk for the kernel to boot with your Linux distribution. If you need to rebuild the kernel of modify some configs you can use:
make linux-2.6-xen-config CONFIGMODE=menuconfig (or xconfig) make linux-2.6-xen-build make linux-2.6-xen-install
This should be the easiest and safest mode to compile a paravirtualized kernel for Xen hypervisor, and should work for most distributions and filesystems images. But for my Ubuntu 9.04 this kernel is a bit old due to udev version and it was quite unstable, the boot log also had some problems and could not boot without assigning the whole RAM memory (specifying the dom0_mem parameter in grub script cause it to crash during boot time with “Out of Memory: Kill process 2 (migration/0) score 0 and children”) and even if I assigned all the physical memory it booted (though it took a very long time) but after a few seconds in gnome desktop it did not respond any more (no input was possible only hard reset).
So as a personal rule I should always use if not the same at least a newer kernel version than the one that came with my Ubuntu 9.04 distribution.
- Vanilla kernel + Opensuse Xen patches
First we need to get the kernel sources for the version we want to patch using Andrew Lyon’s mentained patches for a Xen Dom0 kernel. A good thing to do is to check out Gentoo Xen Patches to see the latest Xen patches version and the vanila kernel we want to download from the Linux Kernel Archive. From Andrew Lyon’s download page we we’ll get the latest xen patches, 2.6.31-7 based on Opensuse’s kernel sources 2.6.31-5. So we do the following:
matmih@Express2:~/Work$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.31.5.tar.bz2 matmih@Express2:~/Work$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.31-7.tar.bz2 matmih@Express2:~/Work$ bzcat linux-2.6.31.tar.bz2 | tar xf - matmih@Express2:~/Work$ cd linux-2.6.31 ; mkdir xen-patches ; cd xen-patches matmih@Express2:~/Work/linux-2.6.31/xen-patches$ bzcat ../../xen-patches-2.6.31-7.tar.bz2 | tar xf - ; cd .. # now apply the pathes ... should be alphabetically ordered in the archive and will only print errors matmih@Express2:~/Work/linux-2.6.31$ cat xen-patches/6*.patch* | patch -p1 -s -l
You now have a possible correct kernel environment that you can configure, build and install for a Dom0 image.
- Xen paravirt_ops Linux kernel
A new kernel infrastructure that allows it to run paravirtualized on a hypervisor, like Xen, VMWare’s VMI and lguest. The infrastructure, supporting x86_32, x86_64 and ia64 architectures, allows you to compile a single kernel binary which will either boot native on bare hardware (or in hvm mode under Xen), or boot fully paravirtualized in any of the environments you’ve enabled in the kernel configuration. Xen pv_ops (domU) support has been in mainline Linux since 2.6.23, and is the basis of all on-going Linux/Xen development. In xen 3.5 it will be the default paravirtualized Dom0 kernel that comes with the hypervizor package. Although we are using the stable version of xen 3.4 we are going to use this kernel as it may as well be the most common way of building a Dom0 kernel in the future. You can find more information at XenParavirtOps page. First we need to get the latest sourcing unsing git found in git-core package:
matmih@Express2:~/Work$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen$ matmih@Express2:~/Work$ cd linux-2.6-xen matmih@Express2:~/Work/linux-2.6-xen$ git pull
Now that we have a valid source kernel environment, made with either methods, though I recommend the last one, we should begin configuring the kernel to be build for your image and hardware requirements, as well as for the requirements of a paravirtualized kernel that can run as a Dom0 kernel, or even in a DomU guest partition. Normally you should copy the kernel configuration that came with your distribution (found in /boot/config-2.6.28- 11-generic for my 64bit Ubuntu 9.04) to linux-2.6-xen/.config and start from there addind Xen features as you find them, as well as the devices may not be included, either in the binary or as modules (use lspci command to see which drivers you need on your machine). If you are using the paravirt-ops kernel, as I did the only thing that you must enable is Dom0 support, so:
matmih@Express2:~/Work/linux-2.6-xen$ make menuconfig # and enable the following # Processor type and features ---> # [*] Paravirtualized guest support ---> # [*] Enable Xen priviledged domain support (NEW)
And the build and install the kernel image. Do not forget to also build the initial ramdisk that will contain some of the modules required for Dom0 to boot.
matmih@Express2:~/Work/linux-2.6-xen$ make
matmih@Express2:~/Work/linux-2.6-xen$ sudo make modules_install install
matmih@Express2:~/Work/linux-2.6-xen$ ls /lib/modules ; ls /boot # to see what was the name of the kernel newly installed
matmih@Express2:~/Work/linux-2.6-xen$ sudo mkinitramfs -o /boot/initrd-2.6-xen.img 2.6.31.5
You should now have the kernel image and initial ramdisk located in your local /boot directory. Also, expecially if you have a SATA disk drive, do not forget to force adding the following modules to your image, if not already built into the kernel image: aacraid, sd_mod, scsi_mod. A detailed description about the SCSI system, for 2.4 kernel at least, can be found at here, and an architectural overview of the implementation can be seen bellow:
So basically you must do the following things to make sure that the above modules are in your ramdisk, depending if you are using mkinitrd, or mkinitramfs utility, where 2.6.31.5 is the kernel version you want to create the image for:
matmih@Express2:~$ sudo depmode 2.6.31.5 matmih@Express2:~$ sudo mkinitrd -v -f --with=aacraid --with=sd_mod --with=scsi_mod /boot/initrd-xen.img 2.6.31.5 # or add the modules to mkinitramfs utility configuration file like I did and run the utility command matmih@Express2:~$ cat /etc/initramfs-tools/modules aacraid sd_mod scsi_mod matmih@Express2:~$ sudo mkinitramfs -o /boot/initrd-2.6-xen.img 2.6.31.5
Usually a common error message when the kernel tries to boot is usually “VFS: Cannot open root device “sda1″ or unknown-block(0,0)“. Also if the boot log indicates that it cannot mount the filesystem you should check if you also added (or built in the kernel image) the required filesystem (like ext3). You can force adding this module as well like above I did for the SCSI SATA disk requirements, but running depmod for the installed modules of the built kernel should add the above modules (you can see what depmod generated looking at, for example, /lib/modules/2.6.31.5/modules.dep file used by the initial ramdisk creation utilities to add the modules to the compressed initrd image).
There are several ways of building a paravirtualized Linux kernel that can be used for a Dom0 or DomU.
- Default kernel from xen package.
In xen3.4.1 sources the modified kernel is based on a 2.6.18.8 version. In order to build everything, including downloading the kernel sources you will need to go to xen directory and:
matmih@Express2:~/Work/xen-3.4.1$ make world
Note*: you will need mercurial packages in order to download the kernel.
You can use the default settings for the kernel configuration (including PCI passthrough in order to keep the same virtual <-> physical mappings of the PCI devices between Dom0 and guest partitions). The built kernel image will be placed at dist/install/boot/vmlinuz-2.6.18.8-xen. Also you will need to configure a ramdisk for the kernel to boot with your Linux distribution. If you need to rebuild the kernel of modify some configs you can use:
make linux-2.6-xen-config CONFIGMODE=menuconfig (or xconfig) make linux-2.6-xen-build make linux-2.6-xen-install
This should be the easiest and safest mode to compile a paravirtualized kernel for Xen hypervisor, and should work for most distributions and filesystems images. But for my Ubuntu 9.04 this kernel is a bit old due to udev version and it was quite unstable, the boot log also had some problems and could not boot without assigning the whole RAM memory (specifying the dom0_mem parameter in grub script cause it to crash during boot time with –complete here the warnings —
- Vanilla kernel + Opensuse Xen patches.
First we need to get the kernel sources for the version we want to patch using Andrew Lyon’s mentained patches for a Xen Dom0 kernel. A good thing to do is to check out Gentoo Xen Patches to see the latest Xen patches version and the vanila kernel we want to download from the Linux Kernel Archive. From Andrew Lyon’s download page we we’ll get the latest xen patches, 2.6.31-7 based on Opensuse’s kernel sources 2.6.31-5. So we do the following:
matmih@Express2:~/Work$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.31.5.tar.bz2 matmih@Express2:~/Work$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.31-7.tar.bz2 matmih@Express2:~/Work$ bzcat linux-2.6.31.tar.bz2 | tar xf - matmih@Express2:~/Work$ cd linux-2.6.31 ; mkdir xen-patches ; cd xen-patches matmih@Express2:~/Work/linux-2.6.31/xen-patches$ bzcat ../../xen-patches-2.6.31-7.tar.bz2 | tar xf - ; cd .. # now apply the pathes ... should be alphabetically ordered in the archive and will only print errors matmih@Express2:~/Work/linux-2.6.31$ cat xen-patches/6*.patch* | patch -p1 -s -l
You now have a possible correct kernel environment that you can configure, build and install for a Dom0 image.
- Xen paravairt_ops Linux kernel
A new kernel infrastructure that allows it to run paravirtualized on a hypervisor, like Xen, VMWare’s VMI and lguest. The infrastructure, supporting x86_32, x86_64 and ia64 architectures, allows you to compile a single kernel binary which will either boot native on bare hardware (or in hvm mode under Xen), or boot fully paravirtualized in any of the environments you’ve enabled in the kernel configuration. Xen pv_ops (domU) support has been in mainline Linux since 2.6.23, and is the basis of all on-going Linux/Xen development. In xen 3.5 it will be the default paravirtualized Dom0 kernel that comes with the hypervizor package. Although we are using the stable version of xen 3.4 we are going to use this kernel as it may as well be the most common way of building a Dom0 kernel in the future. You can find more information at XenParavirtOps page. First we need to get the latest sourcing unsing git found in git-core package:
matmih@Express2:~/Work$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen$ matmih@Express2:~/Work$ cd linux-2.6-xen matmih@Express2:~/Work/linux-2.6-xen$ git pull
Now that we have a valid source kernel environment, made with either methods, though I recommend the last one, we should begin configuring the kernel to be build for your image and hardware requirements, as well as for the requirements of a paravirtualized kernel that can run as a Dom0 kernel, or even in a DomU guest partition. Normally you should copy the kernel configuration that came with your distribution (found in /boot/config-2.6.28- 11-generic for my 64bit Ubuntu 9.04) to linux-2.6-xen/.config and start from there addind Xen features as you find them, as well as the devices may not be included, either in the binary or as modules (use lspci command to see which drivers you need on your machine). If you are using the paravirt-ops kernel, as I did the only thing that you must enable is Dom0 support, so:
matmih@Express2:~/Work/linux-2.6-xen$ make menuconfig # and enable the following # Processor type and features ---> # [*] Paravirtualized guest support ---> # [*] Enable Xen priviledged domain support (NEW)
And the build and install the kernel image. Do not forget to also build the initial ramdisk that will contain some of the modules required for Dom0 to boot.
matmih@Express2:~/Work/linux-2.6-xen$ make
matmih@Express2:~/Work/linux-2.6-xen$ sudo make modules_install install
matmih@Express2:~/Work/linux-2.6-xen$ ls /lib/modules ; ls /boot # to see what was the name of the kernel newly installed
matmih@Express2:~/Work/linux-2.6-xen$ sudo mkinitramfs -o /boot/initrd-2.6-xen.img 2.6.31.5
You should now have the kernel image and initial ramdisk located in your local /boot directory. Also do not forget to force adding the following modules to your image, if not already built into the kernel image: aacraid, sd_mod, scs
matmih@Express2:~$ cat /etc/initramfs-tools/modules
aacraid
sd_mod
scsi_mod