Troubleshooting

MySQL encoding

Occasionally, there are issues with the stock MySQL server/client forcing iso-8859-1 encoding where Enomalism expects UTF-8. The easiest solution is to install the PyPI version of the libraries. Just type the following at the command prompt: 

sudo easy_install -U MySQL-python

KVM MAC Address Conflicts

For some reason, some versions of KVM/Libvirt assign the same mac to both the external tap device in the physical machine, and to the virtual ethernet device inside of the VM. The problem arises in that occasionally the kernel will see the packets as being sent from the same MAC as they are arriving in (over the bridge), so they get dropped. This will prevent you from routing to any machine EXCEPT for the parent physical machine. It can also result in DHCP issues, ARP problems, etc...

The ugly solution is to force the MAC addresses of the vnetX in the physical machine to be different from the VM, or vice versa. I am working on a solution to this problem for the core of enomalism, and perhaps chatting with the libvirt devs.

My NASTY Fix: 

#Run from inside the physical machine:
for NET in 1 2 3;do sudo ifconfig vnet$NET down;sudo ifconfig vnet$NET hw ether 00:FF:FF:FF:FF:FF;sudo ifconfig vnet$NET up;done

Broken KVM Networking

First, go here and read: KVM Bridging on UBUNTU site Essentially, for some reason, you need to let the networking scripts know that you are running a bridge, or the bridges will be flaky. Here is an example bridge that WILL work for our packages. Create this, or a variation as your interfaces file in /etc/networking: NO DHCP (shamelessly copied from ubuntu site) 

auto lo
iface lo inet loopback

auto virbr0
iface virbr0 inet static
        address 192.168.0.10
        network 192.168.0.0
        netmask 255.255.255.0
        broadcast 192.168.0.255
        gateway 192.168.0.1
        bridge_ports eth0
        bridge_fd 9
        bridge_hello 2
        bridge_maxage 12
        bridge_stp off

iface eth0 inet static
        address 172.16.5.0
        netmask 255.255.255.0

or with dhcp: 

auto lo
iface lo inet loopback

auto br0
iface br0 inet dhcp
        bridge_ports eth0
        bridge_fd 9
        bridge_hello 2
        bridge_maxage 12
        bridge_stp off

iface eth0 inet static
        address 172.16.5.0
        netmask 255.255.255.0

Unstable Libvirt

Libvirt 0.4.0 seems to have stability issues when used from Enomalism. Libvirt 0.4.1 has none of those same issues AFAICT, so I would recommend upgrading. We provide packages in our downloads area for several platforms, including Centos5 X86_64. Otherwise, download the tarball from http://libvirt.org/ and recompile yourself.

VM Creator Hangs

The default configuration for the VM Creator only allows for 650 meg ISO images to be uploaded and turned into machines. If you need to use much larger ISO images, you will need to update the "server.max_request_body_size = 1024000000" setting in your configuration to slightly larger than the byte size of your image (750 meg, 4.7G, 9.4g, etc). You will need to restart enomalism after updating this setting.

Errors Connecting to Xen

ie:libvir: Xen Daemon error : internal error failed to connect to xend

A simple test to see if this is a problem is to run virsh -c xen:/// from the command line as root. If you get the above error, you have xend-config.sxp problems!

If you are seeing a lot of errors about connecting to Xen (check your info.log in /opt/enomalism2/logs), your Xen may not have the xmlrpc or unix sockets enabled. Alter your /etc/xen/xend-config.sxp to have the following: 

#(xend-http-server no)

(xend-unix-server yes)

#(xend-tcp-xmlrpc-server no)

(xend-unix-xmlrpc-server yes)

#(xend-relocation-server no)

(xend-relocation-server yes)

Xen initrd and Kernel

Many distros have "broken" initrd/kernel images when it comes to para-virtualized machines under Xen. While their standard way of doing things involves packaging a VM with it's own kernel, that method fails to address upgrading the parent Dom0 OS, since the different kernel versions may be are likely to be incompatible. More discussion here: http://zoetrope.speakermouth.com/2008/2/6/centos-5-kernel-2-6-18-53-1-6-el5xen-initrd-missing-xenblk-modules

Most of the time, the kernel itself does not match that of the packages you can download from our site. This is the cause for the "Kernel image does not exist: /boot/vmlinuz-2.6.18-xen" messages seen by some users. In those cases, edit your enomalism config file to contain the lines (replacing the names as appropriate): 

xen.domu_kernel="/boot/vmlinuz-yourkernel-xen"
xen.domu_initrd="/boot/initrd-yourinitrd-xen.img"

This will result in new machines being provisioned with links to the correct kernel.

If you can start Xen virtual machines, but they die after a few seconds, you probably need an initrd that contains xenblk, but no scsi modules. On Centos/RHEL/Fedora, run the following: 

mkinitrd --omit-scsi-modules --with=xennet --with=xenblk --preload=xenblk /boot/initrd-enomalism-xen $(uname -r)

OTOH, if you are running Ubuntu/Debian, the process is more complicated. You need to remove the scsi, lvm, and mdadm modules from the initramfs-tools folders (especially scsi), as per this post on the Ubuntu Forums :

Hi!

I solved the problem as follows:

I removed the "md" and "lvm" scripts (make a backup copy of them!) from /usr/share/initramfs-tools/hooks and /usr/share/initramfs-tools/scripts/local-top on the host operating system (Dom0) and created an own init ramdisk for the guest operating systems (DomU's)

Code:

mkinitramfs -o /boot/initrd-enomalism-xen $(uname -r)

After that you should restore the "md" and "lvm" scripts from your backup, we just wanted them not to be in the init ramdisk for our DomU's, but maybe you need to recreate the init ramdisk for your Dom0 anytime.

Finally adopt your DomU's config files to point to the correct init ramdisk we just created for them (in my case /boot/xenU-linux-2.6.17-6-generic-xen0.initrd.img).

The DomU's are booting now much faster

I hope this saves a lot of people a lot of trouble in the future. Have a nice day.

NOTE: I have been able to run DomU machines without any InitRD changes on Hardy, so you might want to try before building custom INITRD images.

then... 

mkinitramfs -o /boot/initrd-enomalism-xen $(uname -r)

Then edit the xml of the machine to have the following in the <os> section: 

<initrd>/boot/initrd-enomalism-xen</initrd>

It would also be a good idea to edit the config files to contain the following: 

xen.domu_initrd="/boot/initrd-enomalism-xen"