...my place on the web

  • Increase font size
  • Default font size
  • Decrease font size
XEN Cluster

XEN convert HVM to PVM

Convert Trixbox HVM installation to PVM

For some time I have been running an installation of Trixbox on a small mini-ITX computer. One goal when building the XEN cluster was to move Trixbox to a virtual machine. I didn't migrate the old system to a virtual machine, I just decided to go with a fresh install. Installing Trixbox from iso requires the setup of a HVM, but as I wanted to run it as a PVM I had to convert it. I will here describe how I did to convert the HVM installation of Trixbox to a PVM.


I first installed Trixbox as a HVM using the iso file downloaded from www.trixbox.com.
I will not go through how to install Trixbox as a HVM, there are good resources already on the Net for how to install a HVM from ISO or CD/DVD. Just make sure you have a working installation of Trixbox running as a HVM. The creation of the LVMs below assumes that you have a Volume Group called: vg. If you followed my HowTo you should already have that.

Shutdown HVM

Make sure you shutdown your running Trixbox HVM. Login and run:

[trixbox ~]# shutdown -h now 

Disk setup

We need to copy the files from our HVM disk to a file system that will be used by our PVM. We will use the tools kpartx, losetup and rsync to accomplish this. My HVM installation of Trixbox was done to a LVM called /dev/vg/trixbox which were mapped to a DRBD /dev/drbd6.

Install kpartx:

apt-get install kpartx

Create a loop device to your HVM disk, for me I use /dev/drbd6:

losetup /dev/loop0 /dev/drbd6 

Run kpartx to be able to mount individual partitions:

kpartx -av /dev/loop0

My default installation of Trixbox comprised of 3 partitions, one for root "/", one for boot "/boot" and one for swap. Only the root partition will be of interest. In /dev/mapper/ you should now find 3 devices called loop0p1, loop0p2 and loop0p3. On my system loop0p2 was the root partition. You can run:

kpartx -l /dev/loop0 

to check the size of the partitions. The biggest one should be the root.

Now we mount the root partition to /mnt:

mount /dev/mapper/loop0p2 /mnt/

Check with ls that /mnt is really the root partition:

ls /mnt 

Now we configure two LVMs, one called trixbox-disk and one trixbox-swap. Trixbox-disk will be used to copy all data from the root partition of the HVM installation, the partition that is currently mounted to /mnt.

lvcreate -L 10G -n trixbox-disk vg 
lvcreate -L 512M -n trixbox-swap vg 

Next we will format both volumes:

mkfs.ext3 /dev/vg/trixbox-disk
mkswap /dev/vg/trixbox-swap 

Create a temporary folder and mount it to the trixbox-disk LVM:

mkdir /tmp/trixboxmnt 
mount /dev/vg/trixbox-disk /tmp/trixboxmnt

Now we use rsync to copy the whole root partition from the HVM installation to our new trixbox-disk LVM:

rsync -a /mnt /tmp/trixboxmnt

Edit /etc/fstab in the new LVM to reflect it will run as a DomU PVM. /dev/xvda will be our "/" and /dev/xvdb our "swap":

nano /tmp/trixboxmnt/etc/fstab

Below is a copy of my current fstab, I have kept the old values as comments for reference:

/dev/xvda               /                       ext3    defaults        0 1
/dev/xvdb               swap                    swap    defaults        0 0

#LABEL=/                 /                       ext3    defaults        1 1
#LABEL=/boot             /boot                   ext3    defaults        1 2
#tmpfs                   /dev/shm                tmpfs   defaults        0 0
#devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
#sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
#LABEL=SWAP-hda3         swap                    swap    defaults        0 0

Now we will copy the modules for our current xen kernel to the Trixbox PVM. Replace this with your current kernel:

rsync -a /lib/modules/2.6.24-19-xen/ /tmp/trixboxmnt/lib/modules/2.6.24-19-xen/ 

Unmount /mnt and /tmp/trixboxmnt

umount /mnt
umount /tmp/trixboxmnt

XEN configuration file

Create a xen config file for the Trixbox PVM DomU.

nano /etc/xen/trixboxpvm.cfg

Below is the output of my config file:

name = "trixbox"
memory = 512
root="/dev/xvda ro"
disk = [ 'phy:/dev/vg/trixbox-disk,xvda,w', 'phy:/dev/vg/trixbox-swap,xvdb,w' ]
vif = [ 'mac=00:16:3e:aa:35:32' ]

Try to boot the new Trixbox PVM DomU: Cool

xm create trixboxpvm.cfg -c 

Issue with network

When the Trixbox DomU booted I had an issue with the network, eth0 did not start. Inside the DomU I had to edit the /etc/sysconfig/network-scripts/ifcfg-eth0 file and configure it as below:

[trixbox ~]# nano /etc/sysconfig/network-scripts/ifcfg-eth0 

Restart the network:

/etc/init.d/network restart 

Hopefully the Trixbox DomU is working and is now running as a PVM.

Additional information

You can now go ahead and configure the PVM for DRBD if you have a xen cluster. Check the DRBD section in the xen cluster HowTo: http://www.asplund.nu/xencluster/xen-cluster-howto.html

I think there are basically three ways to run the kernel of the converted HVM.

  1. Use default Ubuntu Dom0 XEN kernel as I did above
  2. Use XEN kernel provided for CentOS and copy it to the Dom0
  3. Use XEN kernel provided for CentOS and use pygrub to boot the DomU

I have only tried #1 and #2 myself as #3 uses pygrub which doesn't run well with DRBD. My preference was to go with #2 as that will keep the original kernel and modules from the Trixbox installation. When doing it that way I could perfectly run the DomU, but I ran into trouble when trying to shutdown and migrate it. It was always complaining about issues to release the two disk devices. Before copying the kernel and initrd to the Dom0 I had to create a new initrd from within the HVM that loads the xennet and xenblk modules, I did that with this command:

[trixbox ~]# mkinitrd /boot/initrd-2.6.18-92.1.6.el5xenU2.img 2.6.18-92.1.6.el5xen --with=xenblk --with=xennet --preload=xenblk --preload=xennet

As I couldn't resolve the issues with #2 I went with #1. I am not 100% sure this is good solution, but at least it is working. Wink The only issue that I notice is a failure during boot to load one module, but no side effects found so far.

Note: Some of the steps above are taken from my memory as I lacked proper notes. I plan to do another test install when time allows. Please let me know of any discrepancies. 

Last Updated on Sunday, 07 September 2008 10:04

XEN extend DomU disk

Extend DomU's LVM/DRBD disk

I had a need to extend one disk in a DomU I use as a file share. This disk is configured for the DomU in addition to the root and swap. This disk is mounted in the DomU and used as the NFS and Samba root. Before I extended this disk I used a test DomU to perform the same operation, on that test DomU I extended the root partition to see if that would cause any problems.

I was successful in all my tests and the only downtime during this operation was a reboot of the DomU. So extending any partition matching the prerequisites below should work without problems.

Below I will described how I went along.


You need to have a DomU installed as described in my XEN cluster HowTo. The DomU will have a separate DRBD device for each file system with the underlying device being a Logical Volume (LV). So for a system with a root and a swap partition, you would have two DRBD devices in the Dom0 each with an underlying LV. So no partitioning should be done in the DomU.


We start by extending the underlying Logical Volume (LV) on both nodes. Run the following (on both nodes):

lvextend -v -L+100G /dev/vg/test-disk 

This will extend the LV called "test-disk" in the Volume Group "vg" with 100GB.

Next step is to extend the corresponding DRBD device so it knows about the increased size of the LV. This step is only required on one node.
Run the following only on the primary node:

root@ha1:/# drbdadm resize test-disk

This will resize the DRBD device called test-disk. Note that I use the same name for the LV as for the DRBD device.

Reboot DomU:

root@ha1:/# xm console test
test:/# reboot

After reboot we need to resize the file system. My device is called /dev/xvdc and it had an original size of 100GB, so I resized /dev/xvdc to 200GB. Login to the DomU again and perform the resize:

root@ha1:/# xm console test
test:/# resize2fs /dev/xvdc 200G

Or at least now you only have to wait for the resize to complete. Please be patient because it can take some time. My resize from 100GB to 200GB took around 10-15 minutes.

Below you fill find the output from "df -h" before and after the resize as well as the output of the resize2fs command:


Before RESIZE:

test:/# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda             9.9G  1.8G  7.6G  19% /
tmpfs                 193M     0  193M   0% /lib/init/rw
udev                  193M   44K  193M   1% /dev
tmpfs                 193M  4.0K  193M   1% /dev/shm
/dev/xvdc              99G   91G  3.1G  97% /data

Performing the RESIZE:

test:/# resize2fs /dev/xvdc 200G
resize2fs 1.40-WIP (14-Nov-2006)
Filesystem at /dev/xvdc is mounted on /data; on-line resizing required
old desc_blocks = 7, new_desc_blocks = 13
Performing an on-line resize of /dev/xvdc to 52428800 (4k) blocks.
The filesystem on /dev/xvdc is now 52428800 blocks long.


test:/# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda             9.9G  1.8G  7.6G  19% /
tmpfs                 193M     0  193M   0% /lib/init/rw
udev                  193M   44K  193M   1% /dev
tmpfs                 193M  4.0K  193M   1% /dev/shm
/dev/xvdc             197G   91G   97G  49% /data
Last Updated on Saturday, 10 October 2009 14:22

Page 2 of 2