www.asplund.nu

...my place on the web

  • Increase font size
  • Default font size
  • Decrease font size
Comments (46)
Problem with auto
1 Monday, 23 September 2013 13:48
Rafal
Great tutorial. But i have one problem. I've installed it on Debian Wheezy and almost all is fine, but ln -s to /etc/xen/auto not working properly. I have one link in HA1 and two in HA2. All is fine when HA1 is down and all is on HA2. But when HA1 is back again i still has all on HA2 Sad Not back. Maybe some auto problem? I tried found an don't your fix, but i don't know where Sad Could you help me with that?
Ganeti : Another solution for clustering with HA
2 Monday, 21 January 2013 22:32
Ganeti is another useful tool for xen clustering with HA features. A web interface allows management of the cluster and VM creation.
Re: pull out the cable from the primary
3 Saturday, 15 December 2012 12:01
Daniel
Hi Felipe,

Honestly I stopped using heartbeat after a system update that broke my configuration and I haven't been bothered redoing it. If I would implement HA again I would probably opt for HA for services running on the VM instead of HA for the VM itself. But of course that would put other requirements on the underlying storage.

A live migration when there is a failure will never work, as the source is dead there is no where to migrate from. So what should happen is that the second host will detect the failure and start the VM from fresh. But I guess that is what you meant. Very Happy

I don't have any additional information that can be of help unfortunately. If I find time I will try to configure HA again, and if I keep my current setup I will do it properly including stonith.
Re: Live migration
4 Saturday, 15 December 2012 11:27
Daniel

Hi Digvijay, Not sure how far you came with your installation reading your post, but it sounds like you have installed and can properly boot the XEN kernel. What method did you try to install a domU? I have always used xen-tools that came with the xen packaged on ubuntu but you could also look at virt-install: http://wiki.xen.org/wiki/DomU_Install_with_Virt-Install

pull out the cable from the primary
5 Wednesday, 05 December 2012 23:05
Felipe Oliveira Gutierrez
Hi Daniel,

Thanks for the tutorial! It is really good!
I am trying to do live migration when I pull out the cable from the master. As others people said, this is not working. I believe I need to use pacemaker with stonith. I am finding it too hard to configure pacemaker. Do you have some links that could help?

Thanks
Felipe
Live migration
6 Tuesday, 04 December 2012 14:45
Digvijay
Hello, I want to do TCP/UDP analysis on xen using live migration of virtual machine. I installed fedora17 on 2 PCs and then installed xen. But now what to do I am not getting enuf idea. So pls help. How to install domU on host and what to do ?
live migration
7 Wednesday, 30 May 2012 09:38
darya

hi i need your help. i wanna do live migration, for this i try to install centos6 on 2 pces, and then install xen on one of them , is it right until now? and after that i don't know really what should i do?

Re: Heartbeat launches both domUs
8 Friday, 06 April 2012 09:45
Daniel
I'm afraid I can't be of any help as I haven't had heartbeat running for a long time. I actually got issues with heartbeat after some system updates in Ubuntu but I never investigated so I don't know the root cause.
So I guess you have to check the logs and trust in google. Wink

Instead I have moved to Debian but still haven't finalised the setup, heartbeat is one outstanding thing. Preparing a tutorial for this new setup.
Heartbeat launches both domUs
9 Thursday, 05 April 2012 23:50
This tutorial has been very helpful. With some tribulations, I've got 2 nodes of xen ubuntu 11.10 dom0s, drbd, and live migration works great.

I have 2 VMs right now, I can migrate them at will.

Heartbeat though... it starts both my VMs on both nodes! Any ideas?
Tremendous Job
10 Saturday, 14 January 2012 20:58
Its a marvellous how-to...

Very descriptive and cut to cut...

Thank you very much...
sudo su
11 Sunday, 24 April 2011 17:49
Ben

"sudo su"?! Surprised What have you done, my eyes are burning!! Use "sudo -i" instead. Rolling Eyes

Very Helpful HowTo for Xen-Cluster! Thank you!
12 Friday, 12 February 2010 10:17
Hansi H

Here you'll find too some helping advices, but not so detailed as the howto here! http://www.thomas-krenn.com/de/wiki/Kategorie:Xen Go cluster! Laughing

Added layer of iSCSI
13 Friday, 27 November 2009 00:14
Lennart Rolland

Hi! I discovered your great project just after starting to build my own xen cluster setup similar to yours. The only difference is that my storage is on a separate storage cluster (two identical servers with software raid + DRBD + LVM + Enterprise iSCSI Target + Heartbeat) and the number of xen dom0 nodes is 9 instead of 2 (11 coputers in total). I choose iSCSI as the storage interface because it is very scalable and a "safe choice" since there are a lot of large companys selling huge hardware iSCSI enabled SAN solutions. So when i grow out of my current homebuilt iSCSI san i can seamlessly migrate to a more powerful solution. I really love XEN becuase it seems to have everything I could possibly dream up. For example XEN supports live migration with iSCSI. Lovely! Anyways if you are interrested here is a little bit of info on how to set up booting your XEN Dom0 and DomUs from iSCSI with Debian Lenny: http://www.etherboot.org/wiki/sanboot/debian_lenny_iscsi I went though wuite a bit having that work in my config and i edited the wiki while workinf so its a bit messy. But it should be complete! PS: The XEN spesific info is at the bottom of the article. Good luck with your project!

xm migrate fails
14 Monday, 09 November 2009 23:48
linux n00b
when I get to the xm migrate step I receive the following error
Error: /usr/lib64/xen/bin/xc_save 22 1 0 0 1 failed

when I cat /var/log/xen/xend.log I see the same error on the primary and pretty much the same on the secondary
XendError: /usr/lib/xen/bin/xc_restore 16 3 1 2 0 0 0 failed

an xm save returns the same error. Any thoughts on where to look?
Stonith
15 Wednesday, 07 October 2009 10:12
vlad
Hi!
Thanks for your guide!
I have an issue:
When I take out the lan cable on first node vm's start very well on the second node, but they are starting on the first node too, because he considers himself primary. If I connect the cable i get duplicates of the vm's running on second node.
What can I do to that would first node shutdown or shutdown heartbeat and drbd that I could to start them manual?
Two primaries
16 Wednesday, 30 September 2009 13:42
Stas

Hi. Super help-full explanation! I followed article closely, and have only one issue - when I start the heartbeat on both servers, I'm getting the VM running on both machines - and the DRBD switches to both primary mode. Does anyone has an idea about this - or can provide a 100% working xendomains scripts? Thanks!

Script for adding domU's ?
17 Wednesday, 10 June 2009 14:02
Hi and greetings from Sweden!

I've successfully used the guide above and have it setup and running. However i wonder if anyone have a nice script or workflow to easy add new domU's to the config?

Hälsningar /Johan
Re: Nice howto, here is a fix
18 Monday, 23 February 2009 11:03
Daniel
Kim, this is covered in the following bug report: https://bugs.launchpad.net/ubuntu/+source/xen-3.2/+bug/216761 Haven't added this in the main HowTo but the link was provided by Frederico above. Probably I wait to update the HowTo until I perform an upgrade to Ubuntu 8.10
Nice howto, here is a fix
19 Tuesday, 20 January 2009 21:18
first great howto, i did as many other and run apt-get upgrade first and that broke a script, well here is a quick fix


###CUT###
--- xendomains 2008-06-04 21:21:55.000000000 +0200
+++ xendomains.fix 2008-06-04 21:23:06.000000000 +0200
@@ -183,7 +183,7 @@
{
name=`echo "$1" | cut -d\ -f1`
name=${name%% *}
- rest=`echo "$1" | cut cut -d\ -f2-`
+ rest=`echo "$1" | cut -d\ -f2-`
read id mem cpu vcpu state tm
All scripts Idea
20 Sunday, 04 January 2009 11:26
Daniel
Even though I have listed all modification to the default scripts it might be a good idea to list all scripts in full. I will make a section for that and post it when I get some time over.

Cheers, Daniel
Re: vm running on both nodes (primary/primary) after reboot
21 Monday, 29 December 2008 23:39
Federico Fanton
Sure, here's the version I'm using -> http://pastebin.com/m3c9625a0
Re: Re: vm running on both nodes (primary/primary) after reboot
22 Friday, 26 December 2008 18:52
Stephan
Thanks for your answer! I think you are right with your guess that something is wrong with the xendomains* script. I found another post where somebody else has exactly the same problem. The problem was that i updated everything before this howto and now the xendomain script is somehow buggy.

May I ask for posting a running xendomains* script or sending it per email (stephanheck'AT'gmx'dot'de)? Then I could test if thats the problem.

Thanks!
Stephan
Re: vm running on both nodes (primary/primary) after reboot
23 Thursday, 25 December 2008 11:57
Federico Fanton
I'd try looking for clues at /var/log/ha-debug on both nodes, maybe xendomains* is getting something wrong :/
vm running on both nodes (primary/primary) after reboot
24 Tuesday, 23 December 2008 23:28
Stephan
First of all thx for the great howto - and merry chrismas Wink

I just followed your how-to line by line and everything seems to be perfect, but after a little bit more testing I run into a problem:

If I reboot ha1 (with vm test running on it), the vm will be migrated correctly to ha2. But when ha1 comes back up it starts a SECOND vm test and drbd says primary/primary.

Do you have a clue what may be wrong.

Thanks in advance!
Stephan
Re: Another xendomains bug
25 Thursday, 04 December 2008 23:31
Federico Fanton
I think I found another bug.. If you put all your VMs inside one of the /etc/xen/auto/* dirs and leave the other one empty, during failback $NAMES (line 333) becomes empty and the script throws a syntax error (resulting in unintended migration of the VMs, in my case Wink )
I patched the scripts and wrote to xen-devel about it.
Why not LVM on DRBD?
26 Monday, 10 November 2008 20:58
Nathan Stratton
Why not run LVM on DRBD? With this method you need to build a DRBD config for every DomU, if you move LVM up a notch you don't need to worry about that. Did you find your method to be faster? I am currently running LVM on DRBD in production for BlinkMind, http://www.blinkmind.com The only downside I found to running LVM on DRBD is the 4 TB limit.

-Nathan

Re: Another xendomains bug
27 Tuesday, 21 October 2008 08:03
Federico Fanton
Oops, it's the same problem, you're right Embarrassed Anyway I tested the patch yesterday, works nicely.
Re: Another xendomains bug
28 Monday, 20 October 2008 19:04
Daniel
I believe you are referring to the same problem as above.
But this patch looks very neat and way simpler than mine. Did you try it and can confirm it works?
Another xendomains bug
29 Monday, 20 October 2008 10:06
Federico Fanton
In case of more than one DomU with XEN_DOMAINS_AUTO, be sure to apply the patch at http://xenbits.xensource.com/xen-unstable.hg?diff/01c8ccb551b0/tools/examples/init.d/xendomains otherwise Heartbeat won't be able to migrate VMs during "xendomains stop", and you'll end up with the VMs on both nodes Crying or Very Sad
Re: Expected behavior on primary node failure
30 Friday, 17 October 2008 09:00
Federico Fanton
Well I tried setting up dopd to prevent splitbrains, but it didn't work as expected.. Maybe I did something wrong, I didn't investigate the matter because I had already spent a lot of time to build the cluster :/ So for the moment I extended deadtime to one minute and wrote notes to *watch out* in case of an unplugged cable.. When there's more time I'd like to try to bind to xendomainsX a script that would check if the actual node is the off-the-net one (by pinging a router maybe) and then shutting down the VMs..
Re: Expected behavior on primary node failure
31 Thursday, 16 October 2008 14:15
Daniel
Did you improve you configuration to prevent this state?

This is something I need to look into myself, but I have had no time so far. I'm currently not using fencing like STONITH.
Re: Expected behavior on primary node failure
32 Thursday, 16 October 2008 07:29
Federico Fanton
I figured it out (by asking on the drbd ML actually Mr. Green ).. What I had when I reattached the cables was a split-brain situation, that's why drbd couldn't resynchronize Smile Thanks all the same!
Expected behavior on primary node failure
33 Tuesday, 14 October 2008 17:33
Federico Fanton
I'm sorry, what's the expected behavior on primary node failure, with your setup? For example, imagine the following:
I have a Samba server on a VM, I pull the network cables from the primary node, and the VM starts on the secondary.. After a while the Samba server comes up again, and I copy a file to it. Then I re-attach the cables.
From my tests, heartbeat shuts the VM down on node2 and restarts it on node1, while DRBD goes in StandAlone mode on both nodes..
What should I do now to prevent losing the file that I copied during node1 failure?
Many thanks for your time, I'm really a HA-newbie Embarrassed
Re: relocation
34 Tuesday, 14 October 2008 17:27
Federico Fanton
As I understand it, live migration is a maintenance/balancing tool, not a high-availability one.. So everything must be in place for it to work, no pulled cables Smile
re: relocation
35 Monday, 08 September 2008 15:59
Paras Pradhan
Yes manual migration is working fine. Not lively migrated when rebooted and shutdown ha1. I have checked haresources and associated files at resources.d and /etc/default/xendomains, all of them are equipped with --live option.

Can anyone tell me one more thing. Is the automatic live migration of Virtual machine possible from ha1 to ha2 if I pull the network cable immediately from ha1?

Thanks
Paras.
Re: Errors in xendomains script
36 Monday, 08 September 2008 08:07
Federico Fanton
I didn't try with more than one DomU, but I had to apply the patch because of many scripting errors :/
Re: Kernel panic on link failure
37 Monday, 08 September 2008 08:03
Federico Fanton
I solved the crashing problem by sheer luck Mr. Green I changed the NIC (I had a 3Com SOHO100TX, switched with a Realtek RTL-8169) and the problem went away..
Many thanks for your helpfulness Smile
Re: relocation
38 Friday, 05 September 2008 23:22
Daniel
Look in your /etc/defaults/xendomains file(s) and check your line with: XENDOMAINS_MIGRATE=
You need to have --live added to that string, like this:
XENDOMAINS_MIGRATE="ha1 --live"

I see now, when looking at this line of the article using Firefox, it looks like a single long dash "-" instead of two dashes. If a manual live migration is working I guess this is your problem, or maybe your /etc/ha.d/haresources file is pointing to the wrong resource-file.
relocation
39 Friday, 05 September 2008 23:01
paras
Instead of live migration, relocating is going on when I reboot the primary node. Paras.
Re: Errors in xendomains script
40 Friday, 05 September 2008 22:14
Daniel
I think I ran that patch on my system as well, but I didn't have any notes to confirm if I did or not. Should add that to my instructions. But that patch is not resolving the issue described above. Did you confirm if you have the same behavior when running more than two DomUs?
Errors in xendomains script
41 Friday, 05 September 2008 21:37
Federico Fanton
For reference, there's a bug report on Launchpad for xendomains: https://bugs.launchpad.net/ubuntu/+source/xen-3.2/+bug/216761
Re: Kernel panic on link failure
42 Friday, 05 September 2008 21:34
Daniel
I followed your steps, but that didn't cause my system to crash. When I checked the logs I found out that crossover link didn't go down as I had Wake on LAN enabled. So I did another test where I disconnected the crossover cable just after shutdown of the master node. But I still did not experience a crash. Let me know if you want to try something else.
Re: Kernel panic on link failure
43 Thursday, 04 September 2008 13:26
Federico Fanton
I'm running Ubuntu 8.04 Server, but the 32bit version.. Could you please try a link failure with your setup?
Steps to reproduce the crash on my system:
- Shutdown -h on the "master" node
- Wait a few minutes
- Panic!

If I ping the vm during the shutdown phase, I get just a 4-seconds gap before the connection is up again (until the kernel crashes, of course) so I think everything is set up correctly.
Many thanks for your help!
Re: Kernel panic on link failure
44 Wednesday, 03 September 2008 18:09
Daniel
I have no issues with kernel panics as described in the bugreport at normal reboots. Have still to do some more extensive testing with heartbeat and effects of certain failures and also I am not sure the DRBD config is optimal.

Are you also running Ubuntu 8.04 Server 64 bit or something else?
Kernel panic on link failure
45 Wednesday, 03 September 2008 15:58
Federico Fanton
Hi! Thanks for your guide, it helped me greatly Mr. Green Anyway, didn't you have any issue with kernel panics on ethernet link failure? (Due to kernel 2.6.24.. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=399981
THANKS
46 Friday, 08 August 2008 12:20
kilolima
Got the link from the ubuntu forums. Thank you very much. Tried it out and it works really nice. Only had one little problem, but that was my fault Smile

Add your comment

Very HappySmileWinkSadSurprisedShockedConfusedCoolLaughingMadRazzEmbarrassedCrying or Very SadEvil or Very MadTwisted EvilRolling EyesExclamationQuestionIdeaArrowNeutralMr. GreenGeekUber Geek
Your name:
Your website:
Subject:
Comment: