Friday, September 02, 2005

Your own mainframe!

Yesterday I worked with Xen for the first time. Only two words truly apply: holy wow!.

For those that haven't seen it before, Xen is a suit of programs and kernel hacks that give you the ability to run multiple installations on one machine simultaneously. In simpler words: one can have the logical equivilant of a mainframe on commodity computers.

This ability is supposedly similiar to other technologies such as UserMode Linux, VMWare and Bochs. I've never used these due to various other problems. I found UML too difficult to get working on Ubuntu. Bochs is supposedly full emulation and runs slowly. VMWare not only supposedly runs poorly as well, but is proprietary to boot. These problems no longer matter to me as I've completely fallin in love with Xen.

This post is about how I got Xen running. Getting to this point involved taking a few wrong turns. If you go about it this way, you shouldn't have nearly the problems that I did.

The first step is to get your hands on a machine that is not too old. I personally would suggest nothing less than a post-2003 machine with no less than 512 megs of ram, 2Ghz processor and an empty 60 gig hard drive. Older machines may work, but my first attempt on a 700Mhz machine with 256 megs of ram failed miserably as the machine's old ACPI setup did not work well with Xen.

I also highly recommend using LVM, as you're going to want to create and delete a lot of partitions without bringing down the machine. Though the Xen docs suggest using 64 megs of memory for the Dom0 superserver, I recommend 256 megs of memory. Otherwise, the system is likely to crash on you when you do any sort of significant LVM operation.

The next step is to install a bare bones linux distro on the machine. You can use whatever you want here. I chose Ubuntu. While installing ubuntu, I set up my partitions thusly:

hda1 ext3 /boot 2 gigs (room for problems)
hda2 ext3 / 2 gigs (1.5 gigs extra)
hda5 fs0 LVM 50 gigs
hda6 fs0 LVM 50 gigs
hda7 fs0 LVM 50 gigs
hda8 fs0 LVM 50 gigs

This setup isn't optimal. Unfortunately, I found out the hard way that Xen apparently has problems with root filesystems on LVM. Thusly, I ended up using hda2 for the root filesystem and put swap in the LVM instead. Some day I'll probably resize hda1 and hda2 to make room for swap.

I installed the Ubuntu server with the "server" option. This started with a reasonably stripped down system. I then downloaded the Xen unstable binary package from the xen home page, which installed its own kernel and modules. I then proceeded to strip out just about anything that wasn't required for system boot or basic system administration. I even took out a few things that required me to remove the "ubuntu-minimal" package which normally requires any packages that are normally required.

This left me with a system that worked fine, but was essentually useless. Just to make sure things were working, I downloaded that minimal linux image that Xen has available for download from sourceforge. Sure enough, after editing a config file, I was able to start these images:

# xm create -c ttylinux-xenconf

The system booted up and everything worked fine. Just for fun, I hacked up a half dozen config files and started several more ttylinuxes.

The time came to get serious. I wanted to base my images off of Ubuntu. As it so happened, the root filesystem was a perfect starting point:

# lvcreate -L1G /dev/fs0 -n bare
# mkfs -t ext3 /dev/fs0-bare
# mount /dev/fs0-bare /mnt
# cp -ax / /mnt; cp /dev/* /mnt/dev
# chroot /mnt
# dpkg --purge (all daemons but postfix + ssh-server,
all modules and kernel stuff, all
non-ext3 filesystem support, etc...)
# umount /mnt

I now had a nice starting image called "bare". I now use this image whenever I want to add a new system to the machine. Think of it as an /etc/skel on steroids:

# lvcreate -s /dev/vs0/bare -n ns2-root -L1G
# lvcreate /dev/vs0 -n ns2-swap -L1G
# mkswap /dev/vs0/ns1-swap

With a beginning image for my nameserver, I then had to set up Xen to know how to start the image:

cat > /etc/xen/auto/ns2 << EOF
kernel = "/boot/vmlinuz-2.6-xenU"
memory = 64
name = "ns2"
nics = 1
ip = ""
gateway = ""
disk = ['file:/dev/fs0/ns2-root,sda1,w']
root = "/dev/sda1 ro"

My virtual server was almost ready. I just had to tune a couple things on the ns2 image to make it ready for boot

# mount /dev/fs0-nfs2 /mnt
# [edit /etc/hostname to reflect]
# umount /mnt

Finally, time to start up the image and make it do something useful:

# xm create -c /etc/xen/auto/ns2
[watch the system boot up]

Great! The system is up. It just so happens that the root account has an ssh key from my laptop, so I ssh in..

# ssh
# apt-get install bind
[ here I also scp'ed in the zonefiles, etc, restarted
bind, etc. Just the same as if it were a completely normal system]

I have one note about the networking: Xen0 bridges connections for the virtual machines. This means that if you set Xen0 up for and put the virtual machine on, then you won't be able to ssh to the machine because they're on different networks. Xen seems to have a problem with routing when ip aliasing is involved.

All said and done, it took me about 8 hours to get this working for the first time. Now that I have things set up correctly, I can make a new server in about 20 minutes.

And that's really basically it. I repeated it a few times for a handful of other machines. Each time I grew a bigger smile on my face as I considered how how much my electric bills are going down.


Blogger Steve said...

By mainframe you mean IBM/AMDAHL/etc mainframe with a z/OS(MVS) operating system?

9:21 AM  
Blogger Macrosoft said...

Great information about mainframe development. Good news.

Mainframe Development

11:44 PM  

Post a Comment

<< Home