This documents the configuration of Xen on Redhat Enterprise Linux 5 based distributions. The choice of Xen over KVM has been made as KVM support in RHEL5 is only a technology preview... maybe we'll switch when CentOS6 is released :)
The following packages must be installed to run xen:
The next step after installing the above packages is to configure the network to cope with virtual machines.
The host network adapter needs to be connected to a bridge which will then have virtual network cards attached to it when VMs are started.
The first stage of this is to create a permenant bridge interface by creating /etc/sysconfig/network-scripts/ifcfg-bridge0 with the following contents:
# /etc/sysconfig/network-scripts/ifcfg-bridge0 DEVICE=bridge0 BOOTPROTO=static TYPE=Bridge # Set this to the MAC of your primary net interface (i.e. eth0) HWADDR=00:01:02:03:04:05 # Set this to the IP the VM host should have IPADDR=123.123.123.123 NETMASK=255.255.255.0 ONBOOT=yes NOZEROCONF=yes PEERDNS=no IPV6INIT=no IPV6TO4INIT=no DHCPCLASS=
It is possible to use DHCP as well, by excluding IPADDR and NETMASK and setting BOOTPROTO=dhcp.
The second stage is to add the host interface into the bridge, assuming that your primary adapter is eth0, this is done by renaming /etc/sysconfig/network-scripts/ifcfg-eth0 to /etc/sysconfig/network-scripts/ifcfg-peth0 and editing it to contain the following contents:
# /etc/sysconfig/network-scripts/ifcfg-peth0 DEVICE=peth0 BRIDGE=bridge0 DELAY=2 BOOTPROTO=none # Set this to the same MAC as bridge0 HWADDR=00:01:02:03:04:05 ONBOOT=yes
This will rename eth0 to peth0 (p to show it is a physical device, not a virtual one) and add it into the bridge created in stage1.
Penultimately edit /etc/grub.conf and change the default= flag to boot the latest xen kernel (the kernel whose name ends in "xen") rather than the stock CentOS one.
Finally enable Xen services and reboot:
chkconfig xend on chkconfig xendomains on reboot
Upon reboot, "ifconfig" should show peth0 instead of eth0, "brctl show" should show bridge0 with peth0 attached and "uname -a" should show the xen kernel running.
By default xen supports dynamic creation of the bridge interface, as this is a permenant xen server, we want to disable this and just use the bridge we created earlier, to do this edit /etc/xen/xend-config.sxp and change the network-script line so that it doesn't do anything as below:
#(network-script network-bridge) (network-script /bin/true)
Now that the host is configured, we can start thinking about configuring the guests.
Firstly we need an install image for the guest OS we want to install. In the case of CentOS5u4 we would run:
mkdir -p /var/lib/xen/images/Centos5u4/x86_64/ wget -O /var/lib/xen/images/Centos5u4/x86_64/vmlinuz "http://www.mirrorservice.org/sites/mirror.centos.org/5.4/os/x86_64/images/xen/vmlinuz" wget -O /var/lib/xen/images/Centos5u4/x86_64/initrd.img "http://www.mirrorservice.org/sites/mirror.centos.org/5.4/os/x86_64/images/xen/initrd.img"
Note that these images are special boot images for xen, most distros should provide a specialised xen boot image for the installer.
Each guest has a configuration file in /etc/xen/ named after the guest hostname, for example a host called mytest should have a configuration file called /etc/xen/mytest.cfg, below is the contents of mytest.cfg:
# xm_vars is defined automatically, use xm_vars.var() to define a variable.
# This function checks whether 'vminstall' has been given a value.
# It is called automatically by 'xm create'.
def vminstall_check(var, val):
try:
val = int(val)
except:
val = 0
if val <= 0:
val = 0
return val
# Define the 'vminstall' variable so that 'xm create' knows about it.
xm_vars.var('vminstall',
use="Reinstall domain if value > 0.",
check=vminstall_check)
# Check the defined variables have valid values..
xm_vars.check()
name = "mytest"
memory = "2048"
vif = [ 'mac=00:16:3e:13:14:15, bridge=bridge0, vifname=%s' % name,
]
disk = [ 'phy:/dev/rootvg/%svol,xvda,w' % name,
]
vnc=0
uuid = "f82138ba-fda7-45cc-a453-273139c3b359"
vcpus=2
if vminstall > 0:
on_reboot = 'destroy'
on_crash = 'destroy'
kernel="/var/lib/xen/images/Centos5u4/x86_64/vmlinuz"
ramdisk="/var/lib/xen/images/Centos5u4/x86_64/initrd.img"
extra="ks=http://mykickstartserver/kickstart.php ksdevice=eth0 text" % name
else:
bootloader="/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'
While this config can seem slightly long at first, most of it is just boiler-plate code for creating an extra flag to install the VM. Anything above name = can be safely left alone. The following items need changing in the configuration above:
Finally, the iptables on the host must be adjusted for each VM. While a host will work without this adjustment, should the iptables service get restarted, the VM will cease to function until xen is restarted. To make the change, add the following line to /etc/sysconfig/iptables, *before* the COMMIT line:
-A Firewall-FORWARD -m physdev --physdev-in mytest -j ACCEPT
Run "service iptables restart" to apply this change (should you want guest IPv6 you may want to do the same with ip6tables, we haven't tested this).
Once the guest is configured, it is very easy to start an installation (for example the mytest VM):
cd /etc/xen/ xm create mytest.cfg vminstall=1
This will start the installation of the VM, "xm list" should show that the VM is running. You can access the installer console by running "xm console mytest" (to exit this press ctrl+] and run "reset"). Once the install is complete, the machine should shutdown and stop. The machine can then be booted for the first time using xm create mytest.cfg. If you want the machine to start when the host machine is booted run:
cd /etc/xen/auto ln -s ../mytest.cfg .
All configs linked into the auto directory will auto-start with the host.
Once the guest machine is installed, it can be treated just like a real host, the only difference being if you shut it down you'll have to login to the host to turn it back on!
This is a very common problem, normally you'll find you have a typo in one of the network-scripts files (i.e. an incorrect MAC address or something). Running "service network restart" may reveal the error. Also check that you have bridge-utils installed, if it is missing nothing will error, but networking just won't work!
This is normally caused by forgetting to create a new logical volume for the VM before attempting to start it. xm create will usually hang for about a minute before displaying this message if there is something wrong.