Sunday, September 18, 2005

The Doldrums of History

In a recent paper I wrote that a revision control system with lossy storage is a possible way to get around the problems that typically affect revision control. The reactions to this were not very surprising, typically reducing to something along the lines of 'Dude, storying all of the past storage is the whole point to revision control!'

Essentially, the RCS field has come to terms with the thought that storing the full history is a basic truth of revision control system. The reasoning that leads to this position is generally pretty pretty good. Several lines of reasoning support this assessment with three of the most important ones being: merging capabilities are affected by the amount of history present, the person responsible for the codebase may come under legal scrutiny current and users have the expectation that anything they store they will be able to retrieve.

This evidence, though it appears to be solid on the surface, is actually circumstantial. In this paper I'll examine the three reasons at length and illustrate that storing full history, while potentially useful, is not actually an inherent truth of revision control systems.

The first supporting reason for lossless revision storage is that merging is affected by the history present. This statement is certainly true with certain implementations; a user attempting to perform a merge with history related to the merge with the RCS system named "tla" is in for a hard row indeed. Other revision control systems, such as Bazaar, are able to partially sidestep the problem by searching for, and usually finding a suboptimal, but fully traceable, history. These two tools have this obligation because they rely upon something called three-way merging. The details behind three way merging are complex, though a oversimplified explanation will suffice for this conversation: A three way merge figures out which patches to apply by finding a previous point in time in which the two branches were identical. The changes between the common ancestsor and the banch to be merged are figured out and then these changes are applied to the other branch.

Other revision control systems, such as bazaar-ng, are seriously discussing avoiding three way merges entirely. The process is a bit different, but presumably also requires the knowledge of what modifications have already been applied. By applying just the patches that are missing, one can perform the equivilant of three way merge without actually performing a merge. This means that finding a common ancestor is no longer obligatory.

The full history is not required to perform these merges. The only fundamental component that is strictly required to perform a three way merge is a record of what has been previously applied so as to avoid applying it again. This does not mean that many revision control systems do not keep the full history (many of them, such as Bazaar, do so by default). This extra information isn't kept strictly to perform a three way merge; rather its kept in order to perform more advanced types of three way merges such as intentionally storing a false copy of accurate history (which I'll henceforth refer to as "a corrupted merge"). In simpler terms, *a record of what history has been seen is required, but not the history itself*

Prior to going into this argument I should clearly point out that I do not qualify as a lawyer, a paralegal or for that matter a filing clerk at the courthouse. The information contained herein is not intended to be legal advice. If you are seeking legal advice then I suggest you seek a lawyer for lawyerly things.

The second listed reason for indefinitely history is that the person maintaining a branch or archive may come under legal scrutiny. A compelling example of this reasoning is a court case (still going on at the time of this writing) named SCO Vs. IBM. Though SCO's arguments are rather unclear, the cental tenant of their argument seems to be that IBM misapropriated source code that belonged to SCO was unlawfully given to the Linux kernel. The Linux community reacted quickly to these charges by scouring the source code for submissions that were potentially infringing. The result of this search is generally considered negative; the public is not able to identify infringing code. This case is very well known, so for the sake of discussion lets assume that infringing code had been found. We'll also assume, again for the sake of argument, that IBM was the proper entity to litigate against.

The assumption that infinging code had beenperson that found makes no difference, as legal responsibility flows upwards. The person ultimately legally responsible for an infringing action is not the person that performed the infringement, but the highest ranking executive (or lead developer for those projects that do not have a formal organization) that had control over the organization's policies. The failure, though created by a typically low ranking individual, should have been caught misapropriating code by policy and workflow monitoring that ensured that such infringement could not happen without being caught eventually. This can be illustrated by examing the policies that Linus Torvalds put into place after the initial SCO allegations; he would be even more guilty than IBM for having first accepted (via delegation), then propogated the infringement. To avoid this sort of repeat Torvalds now requires that each person involved in the chain of merging of a submission sign off on code that it is not infringing.

This clearly illustrates that clearly identifying exactly who contributed what code at what time becomes less important as the longer the infringing code remains in the codebase, the more responsible the maintainer is and the less responsible the submitter. Regardless, its typically
nice to be able to point the finger at somebody else and blame someone else on a moral level, even if legal responsibility can not be evaded. This feel-good act is actually ultimately self-destructive.

The majority of programmers live under legal systems that are very closely related to the British style of judicial reasoning. In these legal systems plausible deniability is a an appropriate legal excuse. Any entity being sued another one looses the ability to destroy any of the entity's history or risk charges of destruction of evidence, tampering with evidence, interfering with an investigation and so on. Organizations will typically evade this sort of problem by intentionally intituting policies that ensure for the destruction of all possible company records after the elapsation of any legally mandated minimums.

These two facts put side by side show that from a legal position indefinitely storing history is not a net win; rather its a drawback. Storing as much history as possible will not provide a legal defense, but will hinder one.

The final listed reason for storing full history is that the user of a revision control system typically expects to be able to pull out by demand the representation of his software at any arbitrary point in the past. Supposedly any revision control system that is not able to perform this action is not a revision control system at all, which can pose interesting questions for whether systems like git is a revision control system at all.

This expectation does not meet with actual practice. The value of any given revision of software is inversely proportional to the distance from the current version. As a case in point, consider which version of his software a developer would rather loose: a version from two weeks ago or a version from two years ago. Additionally, the further back in time a developer looks, the less granularity that a user needs. A user may wish to distinguish the code from today from the code from last thursday, or the code from the beginning of this month from the beginning of two months ago, or today's code from two years ago. However, when users are comparing today's code against code from two years ago aren't generally doesn't care if the the version from two years ago happened on a Tuesday or a Wednesday.

Unfortunately most contemporary revision control systems treat all versions equally. The baggage that comes with storing the third revision alongside the six thousandth revision is for all practical purposes equivilant. Thus, a version of inherently less value costs just as much as current software.

The equal cost for revisions means that as the number of revisions increase, the cost for working with the newer revisions will increase. In many RCSs the definition of the current version is in part defined by the sum of its previous versions with a little bit more added. Different RCSs handle the past history differently; Arch and derivitaves such as Bazaar store this previous history as changesets. Others, such as Bazaar-NG, store the changes on a per-file basis. Regardless, typically all previous revisions are there in some form or another, standing by just in case they are needed by the user or the RCS.

Many revision control systems,including Bzr for the not-most-current case, use a recursive algorithm: To get a particular version of a file or changeset, one applies a set of changes to the previous file or revision, which in turn will apply a set of changes to the previous file or revision. This recursive algorithm is the achilles heel of revision control systems.. The achilles heel shows itself in two ways. The first problem is that in order to use the latest versions one must carry around the older revisions as well. The second problem is that in order to access the latest versions one must access the older versions.

The result of these two problems is that the cost of any current version is the sum of the cost of all previous versions plus a little bit extra. Thusly, as versions are saved, the cost goes up while the worth declines. Remember: any given version has less value as it ages. The resulting inflation, because of the recursive nature, affects the latest versions most strongly.

Many people have been seen stating that a good revision control system solution is similiar to a good backup solution. Surely this premise applies to a revision control system then it would apply to a backup system as well. If these two types of systems are similiar, then why doesn't one see this sort of lossy approach taken with backup systems? The simplest answer is that one does see *exactly this sort of behavior* in some backup systems. The traditional unix tape rotation system is explicitely designed to provide long term storage with a level of looser granularity as you look at older tapes. One will be able to find a tape for a specific day in the last two weeks, a tape for a specific previous week for the last month, a specific month in the last year and so on. Rarely does one find a backup system run by a highly qualified admin that stores a daily version going back to the project's inception (or even daily for the last years). Additionally, many moden revision control systems handle this rotation automatically by systematically pruning data that closely resides next to other older stored data.

Maintaining all previous history is not required. The common reasons for maintaining all history actually provides stronger arguments for tracking history and not maintaining it. This leaves RCS developers with a much more interesting question: When should things be tracked and when should things be maintained?

Tuesday, September 13, 2005

The Achilles heel of DRCS

One of the things that most distributed revision control systems do is that they keep a permanant, indefinite record of everything that's ever happened to a branch. This can be useful if some company in Santa Cruz decides to defame you or your company. Most of the time this older information is unnecessary baggage that is essentially kept around just in case its ever needed.

This is the achilles heel of distributed revision control. As time passes the storage of past events grow indefinitely and become unwieldy to work with. Some tools, such as bazaar-NG, are working to address certain aspects of this.

Whats needed is a way to conflate history by merging together old revisions. By conflating revisions a bit of disk space is saved, network access is reduced and the amount of work involved in building by using ancestors is reduced.

Consider, for example, some sort of pool of patches. If the access to these patches is monitored then one can detect which patches are no longer very interesting. If two non-interesting patches are next to each other, then merge them into a single superpatch.The two patches in the pool can then be removed.

As time passes older, unused patches will continue to clump together. These multipatches can be used in the place of the individual patches in any place that all of them would have been applied anyways.

If the user wants a revision that has been merged into a multipatch, just go ahead and get it and add it to the pool. It can be munged with other patches just as with prior events.

Bazaar-NG Hackfest coming up

A bazaar-NG hackfest is coming up this November. The event is going to be in Toronto, Canada and will be work, work, work. If you like to work your ass off for free and make cool things happen, then I'll meet you there. :)

If you would like more information about the hackfest, then you can read about it at The Ubuntu Below Zero Page.

Sunday, September 11, 2005

New article

Tonight I wrote a new article for This article covers why distributed revision control is a good thing. You can find the article at My first LinuxGazette article

Saturday, September 10, 2005

My first impressions of full ubuntu

I have been using ubuntu since before it was called ubuntu (back then it was no-name-yet). I was also the first person that used KDE in the Ubuntu world, predating the efforts of noble projects such as Kubuntu.

I did this not because I happen to be a great KDE fan, but because the gnome software wasn't capable of properly handling my dual head setup properly. If you've ever run a dual-head setup yourself (this is not only possible but reasonably easy with most laptops), then you'll understand the importance of both maximize only applying to one monitor and having windows be created on the same screen that the mouse pointer happens to be on.

The ACPI team has informed me that they plan on hijacking my laptop, a behemoth Sony Vaio A190, while meeting in Canada. This gave me a great excuse to get another laptop. This new one, a T350P is as small and long lasting as the A190 is large but short lived.

Ironically, the new T350P is even more odd than the A190 This machine runs Ubuntu for approximately six hours on a single charge, has a widescreen 1280x768 display and gobs of ACPI problems, most of which have been fixed in the official Ubuntu.

So far my opinion of gnome on the T350P is rather mixed. Watching the processor speed is almost as nice as being able to control the lcd brightness. With the old laptop I was able to apt-get install just about anything I wanted and have the software show up in K->Debian menu. On the little laptop I haven't figured out how to get unofficial software such as pingus to show up in the gnome menus.

Wednesday, September 07, 2005

Gnuarch firestorm pt. 2

As the gnuarch firestorm continues on... well, you read the link and figure it out. :)

Secretaries sacked after cyber brawl | NSW/ACT | Breaking News 24/7 - (08-09-2005): "'She's a very nice, quiet girl,' Mr Bird said. 'It is just a silly girl thing that got out of control.'"

Individuals in a Capitalistic Society

The gnuarch mailing list is under fire again. Companies are evil this, venture capitalists are slash and burners that.. the same old story from you know who. When I hear tirades such as these, I can't help but consider the motivations of the writer. Is the person complaining that the system does not work at all, or is the individual is complaining that the system does not work for them?

The prototypical industrialized society is framed around the capitalist system. This system provides for the private ownership, either individually or as groups, to conduct commerce interdependantly with a minimal amount of government controls. An entity can focus on the things that it is good at and trade resources (typically capital, workprodcut, goods or ingenuity) for the things that it is not. This specialization allows for much greater efficiencies than any person could achieve individually.

A player in the capitalist model, whether its an inventor, an investor, a small company or a large corporation, is essentially a living organism that requires certain resources to exist. These players, instead of living off of salads and cheeseburgers, live on capital. Any entity in the capitalist model that runs out of capital will starve and die no less surely than any breathing creature that runs out of food.

This means that the capitalist system is, to some extent, darwinistic. Players, in order to successfully compete against other players, will maximize their value (price vs. worth) by making arrangements with the most effciencient players that they have access to. Failure to do so results in less capital as the competiting players become more efficient than the players that don't.

The price of a product is not limited to the initial cost. Direct factors include, but are not limited to: transportation costs, resale value, maintenance costs. Indirect price factors include the cost of maintaining the relationship with the vendor such as, but again not limited to: Effort contacting the vendor, the integrity of the vendor and the cost of maintaining a positive relationship with the vendor.

A key point that many individuals fail to realize is that individuals are mandatory participants. An individual, just the same of a large multi-regional conglomerate, has a certain amount of resources, whether its capital (venture capitalists), workproduct, ingenuity or other things. These resources can be traded for other resources such as a paycheck, partial company ownership, health care, pensions and so forth. These resources are used to keep the indivitual playing in the game, so that they are able to earn (and potentially reinvest) for more capital.

Just as with any corporation, an individual can price themselves out of the market in any of a variety of ways. One of the many ways that an individual can undercompete is asking for an amount of capital that exceeds thier worth as perceived by others. Another, more subtle, way to undercompete is to make the cost in effort of working with the individual high enough that potential worth is overwhelmed. An individual that is unreliable, inconsistant and works poorly with others may not be able to trade his inherent worth at any price (even free).

Thus, any person or company that finds itself running out of capital must immediately improve the offered value of its goods or services or face failure. Failure to do so can result in metaphorical (literal in the case of the individual) starvation. Thusly, a player must reduce either cost or increase worth. Increasing worth over the immediate term is typically not practical. Reducing cost, however is.

For an individual, one can either reduce the price for doing business or reduce the cost of doing business with the individual. An individual faced with the need to immeidately reduce cost has options available to him or her, including: Improving reliability, improving consistancy, improving honesty, improving relations with others and getting along better with others.

Fighting with others, complaining and other non-constructive acts while under the threat of starvation is a short sighted methodology that will not work.

Monday, September 05, 2005

Tracking too many irc channels

As part of my job I have to track a lot of irc channels. Throw on top of the ones that I'm personally interested in, I have to watch a lot of stuff. Theres far too many channels to constantly switch from window to window; one would literally wear out the /win keys on the keyboard. So how does one monitor multiple channels at once?

Simple, really:

  • Turn on logging in irc (I believe that's /log *)
  • Setup a tall console with the smallest font that you can easily read
  • Tail -f /var/log/freenode/*
And thats all there is to it. Now you can easily keep an eye on dozens of conversations at the same time. :)

Live archive support

As many of you are aware, recently the old server suffered massive filesystem corruption (thanks Hans). The machine is back up and configured and is all ext3. The new machine, that used to be called, is now

Reaching the machine via ssh/sftp is a little tricker than before. One can do this by adding the following to their ~/.ssh/config:

Host codehome
Port 1008
ProxyCommand none

Reaching the machine via http is a great deal easier. In that case, one simply points their browser or bazaar at: Listing files will not be necessary as I intend to support readable webdav.

As with before this machine lives on my connection at home. Thusly, if its reasonably possible, use If for some reason won't work for you, get ahold of me and I'll set you up with an account.

For those that it will help, I have a backup of the system from October of 2004, which was the date that the supermirror officially moved to Canonical.

Bazaar-NG Wiki updates

Good documentation is important for any project that wants to grow up strong and healthy. Unfortunately, doing so is just about as tasty as eating vegetables.

I've just finished updating the bzr wiki site a bit. I pulled several things off the mailing list, updated various version numbers, and rewrote the Bzr page.

Getting community stuff going

Now that I finally have a good handle on Xen I'm starting to do some excellent stuff with it. In the space of about five minutes I can get a new host up on the internet.

I had gotten rather cramped hosting the stuff that I was. I'm very happy about being able to make virtual machines that are exactly tuned to one specific thing. Now I don't have to worry about which people have access to which things via which methods. I just make a new machine for each thing. :)

Tonight I also setup a planet for revision control users and developers. You can find it at I also did a bit of tuning at the wiki and setup a wiki which is essentially empty.

If you work on a free software revision control system and have a blog, then I'm looking for you! Shoot me a mail ( will work as well as anything) and I'll make sure to get it added to the planet.

I also plan on setting up rss feeds for other stuff as well, so that I can planetize them as well.

Sunday, September 04, 2005

Poor english

After screwing with xen for the last four hours I decided to wind down with some netstumbler. One of the pages that I came across was: Linux Shell Scripting Tutorial - A Beginner's handbook. As the title states, this is a beginner's book. The quality of the information is actually pretty good once you get past the poor english within the document.

Its people like Vivek that warm my 'hart' on cold winter days. I see so many people that couldn't care less for others. Vivek, despite his english limitations, is clearly going through great personal effort to help others. This effort is admirable. Vivek, your first beer in Pennsylvania is on me.

In closing, I include some poetry from our writer:

I love linux.

It is different from all other Os
My brother Vikrant also loves linux.
He currently lerarns linux.
Linux is cooool.
Linux is now 10 years old.
Next year linux will be 11 year old.
Rani my sister never uses Linux
She only loves to play games and nothing else.
Do you know?
(DOT) is special command of linux.
Okay! I will stop.

Saturday, September 03, 2005

Notes From Inside New Orleans by Jordan Flaherty

Notes From Inside New Orleans by Jordan Flaherty
Friday, September 2, 2005

Thanks to all the loved ones and long-lost friends for your sweet notes of concern, offers of housing and support, etc. Yes, I stayed through the storm and aftermath. I'm fine - much better off than most of my brother and sister hurricane survivors. Below is my attempt to relay some of what I've seen these last few days.

I just left New Orleans a couple hours ago. I traveled from the apartment I was staying in by boat to a helicopter to a refugee camp. If anyone wants to examine the attitude of federal and state officials towards the victims of hurricane Katrina, I advise you to visit one of the refugee camps.

In the refugee camp I just left, on the I-10 freeway near Causeway, thousands of people (at least 90% black and poor) stood and squatted in mud and trash behind metal barricades, under an unforgiving sun, with heavily armed soldiers standing guard over them. When a bus would come through, it would stop at a random spot, state police would open a gap in one of the barricades, and people would rush for the bus, with no information given about where the bus was going. Once inside (we were told) evacuees would be told where the bus was taking them - Baton Rouge,
Houston, Arkansas, Dallas, or other locations. I was told that if you
boarded a bus bound for Arkansas (for example), even people with family and a place to stay in Baton Rouge would not be allowed to get out of the bus as it passed through Baton Rouge. You had no choice but to go to the shelter in Arkansas. If you had people willing to come to New Orleans to pick you up, they could not come within 17 miles of the camp.

I traveled throughout the camp and spoke to Red Cross workers, Salvation Army workers, National Guard, and state police, and although they were friendly, no one could give me any details on when buses would arrive, how many, where they would go to, or any other information. I spoke to the several teams of journalists nearby, and asked if any of them had been able to get any information from any federal or state officials on any of these questions, and all of them, from Australian tv to local Fox
affiliates complained of an unorganized, non-communicative, mess. One cameraman told me "as someone who's been here in this camp for two days, the only information I can give you is this: get out by nightfall. You don't want to be here at night."

There was also no visible attempt by any of those running the camp to set up any sort of transparent and consistent system, for instance a line to get on buses, a way to register contact information or find family members, special needs services for children and infirm, phone services, treatment for possible disease exposure, nor even a single trash can.

To understand this tragedy, its important to look at New Orleans itself.

For those who have not lived in New Orleans, you have missed a incredible, glorious, vital, city. A place with a culture and energy
unlike anywhere else in the world. A 70% African-American city where resistance to white supremecy has supported a generous, subversive and unique culture of vivid beauty. From jazz, blues and hiphop, to secondlines, Mardi Gras Indians, Parades, Beads, Jazz Funerals, and red beans and rice on Monday nights, New Orleans is a place of art and music and dance and sexuality and liberation unlike anywhere else in the world.

It is a city of kindness and hospitality, where walking down the block
can take two hours because you stop and talk to someone on every porch, and where a community pulls together when someone is in need. It is a city of extended families and social networks filling the gaps left by city, state and federal goverments that have abdicated their responsibilty for the public welfare. It is a city where someone you walk past on the street not only asks how you are, they wait for an answer.

It is also a city of exploitation and segregation and fear. The city of
New Orleans has a population of just over 500,000 and was expecting 300 murders this year, most of them centered on just a few, overwhelmingly black, neighborhoods. Police have been quoted as saying that they don't need to search out the perpetrators, because usually a few days after a shooting, the attacker is shot in revenge.

There is an atmosphere of intense hostility and distrust between much of Black New Orleans and the N.O. Police Department. In recent months, officers have been accused of everything from drug running to corruption to theft. In seperate incidents, two New Orleans police officers were recently charged with rape (while in uniform), and there have been several high profile police killings of unarmed youth, including the murder of Jenard Thomas, which has inspired ongoing weekly protests for several months.

The city has a 40% illiteracy rate, and over 50% of black ninth graders will not graduate in four years. Louisiana spends on average $4,724 per child's education and ranks 48th in the country for lowest teacher salaries. The equivalent of more than two classrooms of young people drop out of Louisiana schools every day and about 50,000 students are absent from school on any given day. Far too many young black men from New Orleans end up enslaved in Angola Prison, a former slave plantation where inmates still do manual farm labor, and over 90% of inmates eventually die in the prison. It is a city where industry has left, and most remaining jobs are are low-paying, transient, insecure jobs in the service economy.

Race has always been the undercurrent of Louisiana politics. This
disaster is one that was constructed out of racism, neglect and
incompetence. Hurricane Katrina was the inevitable spark igniting the gasoline of cruelty and corruption. From the neighborhoods left most at risk, to the treatment of the refugees to the the media portayal of the victims, this disaster is shaped by race.

Louisiana politics is famously corrupt, but with the tragedies of this week our political leaders have defined a new level of incompetence. As hurricane Katrina approached, our Governor urged us to "Pray the hurricane down" to a level two. Trapped in a building two days after the hurricane, we tuned our battery-operated radio into local radio and tv stations, hoping for vital news, and were told that our governor had called for a day of prayer. As rumors and panic began to rule, they was no source of solid dependable information. Tuesday night, politicians and reporters said the water level would rise another 12 feet - instead
it stabilized. Rumors spread like wildfire, and the politicians and
media only made it worse.

While the rich escaped New Orleans, those with nowhere to go and no way to get there were left behind. Adding salt to the wound, the local and national media have spent the last week demonizing those left behind. As someone that loves New Orleans and the people in it, this is the part of this tragedy that hurts me the most, and it hurts me deeply.

No sane person should classify someone who takes food from indefinitely closed stores in a desperate, starving city as a "looter", but thats just what the media did over and over again. Sherrifs and politicians talked of having troops protect stores instead of perform rescue operations.

Images of New Orleans?hurricane-ravaged population were transformed into black, out-of-control, criminals. As if taking a stereo from a store that will clearly be insured against loss is a greater crime than the governmental neglect and incompetence that did billions of dollars of damage and destroyed a city. This media focus is a tactic, just as the eighties focus on "welfare queens" and "super-predators" obscured the simultaneous and much larger crimes of the Savings and Loan scams and mass layoffs, the hyper-exploited people of New Orleans are being used as a scapegoat to cover up much larger crimes.

City, state and national politicians are the real criminals here. Since
at least the mid-1800s, its been widely known the danger faced by
flooding to New Orleans. The flood of 1927, which, like this week's
events, was more about politics and racism than any kind of natural
disaster, illustrated exactly the danger faced. Yet government
officials have consistently refused to spend the money to protect this poor, overwhelmingly black, city. While FEMA and others warned of the urgent impending danger to New Orleans and put forward proposals for funding to reinforce and protect the city, the Bush administration, in every year since 2001, has cut or refused to fund New Orleans flood control, and ignored scientists warnings of increased hurricanes as a result of global warming. And, as the dangers rose with the floodlines, the lack of coordinated response dramatized vividly the callous disregard of our elected leaders.

The aftermath from the 1927 flood helped shape the elections of both a US President and a Governor, and ushered in the southern populist politics of Huey Long.

In the coming months, billions of dollars will likely flood into New
Orleans. This money can either be spent to usher in a "New Deal"
for the city, with public investment, creation of stable union jobs, new schools, cultural programs and housing restoration, or the city can be rebuilt and revitalized? to a shell of its former self, with newer hotels, more casinos, and with chain stores and theme parks replacing the former neighborhoods, cultural centers and corner jazz clubs.

Long before Katrina, New Orleans was hit by a hurricane of poverty, racism, disinvestment, de-industrialization and corruption. Simply the damage from this pre-Katrina hurricane will take billions to repair.

Now that the money is flowing in, and the world's eyes are focused on Katrina, its vital that progressive-minded people take this opportunity to fight for a rebuilding with justice. New Orleans is a special place, and we need to fight for its rebirth.

What's Wrong With Sorcerer?

Here's a post from a friend of mine, Dennis. I suspect that he's a bit pissed off about people changing interfaces too quickly. Does anybody else know someone like this? I Hate Blogs: What's Wrong With Sorcerer?

A quote from New Orleans Mayor

Ray Nagin, mayor of New Orleans, had this to say about a week after the great New Orleans flood:

Don't tell me 40,000 people are coming here. They're not here. Its too doggone late. Now get off your *** and lets do something and lets fix the biggest *** crisis in the history of this country

He has a point. As Americans throughout the country mobilize, gather food and water, and send support down to Louisiana one can't help but wonder why its not already there. I have no doubt that if Hollywood had burned down, the cargo jets would have flown, the trucks would have gone rolling and they would have omlettes for breakfast.

Most Recently Eligible Bachelor

I received a call from my lawyer today. I found out that my divorce is finally final. What a mess. Some people say "don't get divorced". I say "don't get married".

On a more positive note, my ex-wife told me that they were collecting water at the local mall for the Louisiana tragedy. Some of those people out there are dying for lack of something to drink. With this thought in mind, I accepted the ex's offer to use her van to carry water.

Unfortunately, the nearby store was nearly out of water. They're entire stock of gallon jugs of water were gone. Fortunately, not many people realize that distilled water is perfectly drinkable. Well, that was until I their last 90 gallons. Now they really are out of water. This tragedy makes the 2005 Wilkes-Barre, PA flood look like the Sahara.

On a more trivial note, I noticed that the Better Business Bureau hides the Microsoft complaints(normally they list how many complaints were made, how many were resolved, how many weren't). I wonder why that is?

Friday, September 02, 2005

Your own mainframe!

Yesterday I worked with Xen for the first time. Only two words truly apply: holy wow!.

For those that haven't seen it before, Xen is a suit of programs and kernel hacks that give you the ability to run multiple installations on one machine simultaneously. In simpler words: one can have the logical equivilant of a mainframe on commodity computers.

This ability is supposedly similiar to other technologies such as UserMode Linux, VMWare and Bochs. I've never used these due to various other problems. I found UML too difficult to get working on Ubuntu. Bochs is supposedly full emulation and runs slowly. VMWare not only supposedly runs poorly as well, but is proprietary to boot. These problems no longer matter to me as I've completely fallin in love with Xen.

This post is about how I got Xen running. Getting to this point involved taking a few wrong turns. If you go about it this way, you shouldn't have nearly the problems that I did.

The first step is to get your hands on a machine that is not too old. I personally would suggest nothing less than a post-2003 machine with no less than 512 megs of ram, 2Ghz processor and an empty 60 gig hard drive. Older machines may work, but my first attempt on a 700Mhz machine with 256 megs of ram failed miserably as the machine's old ACPI setup did not work well with Xen.

I also highly recommend using LVM, as you're going to want to create and delete a lot of partitions without bringing down the machine. Though the Xen docs suggest using 64 megs of memory for the Dom0 superserver, I recommend 256 megs of memory. Otherwise, the system is likely to crash on you when you do any sort of significant LVM operation.

The next step is to install a bare bones linux distro on the machine. You can use whatever you want here. I chose Ubuntu. While installing ubuntu, I set up my partitions thusly:

hda1 ext3 /boot 2 gigs (room for problems)
hda2 ext3 / 2 gigs (1.5 gigs extra)
hda5 fs0 LVM 50 gigs
hda6 fs0 LVM 50 gigs
hda7 fs0 LVM 50 gigs
hda8 fs0 LVM 50 gigs

This setup isn't optimal. Unfortunately, I found out the hard way that Xen apparently has problems with root filesystems on LVM. Thusly, I ended up using hda2 for the root filesystem and put swap in the LVM instead. Some day I'll probably resize hda1 and hda2 to make room for swap.

I installed the Ubuntu server with the "server" option. This started with a reasonably stripped down system. I then downloaded the Xen unstable binary package from the xen home page, which installed its own kernel and modules. I then proceeded to strip out just about anything that wasn't required for system boot or basic system administration. I even took out a few things that required me to remove the "ubuntu-minimal" package which normally requires any packages that are normally required.

This left me with a system that worked fine, but was essentually useless. Just to make sure things were working, I downloaded that minimal linux image that Xen has available for download from sourceforge. Sure enough, after editing a config file, I was able to start these images:

# xm create -c ttylinux-xenconf

The system booted up and everything worked fine. Just for fun, I hacked up a half dozen config files and started several more ttylinuxes.

The time came to get serious. I wanted to base my images off of Ubuntu. As it so happened, the root filesystem was a perfect starting point:

# lvcreate -L1G /dev/fs0 -n bare
# mkfs -t ext3 /dev/fs0-bare
# mount /dev/fs0-bare /mnt
# cp -ax / /mnt; cp /dev/* /mnt/dev
# chroot /mnt
# dpkg --purge (all daemons but postfix + ssh-server,
all modules and kernel stuff, all
non-ext3 filesystem support, etc...)
# umount /mnt

I now had a nice starting image called "bare". I now use this image whenever I want to add a new system to the machine. Think of it as an /etc/skel on steroids:

# lvcreate -s /dev/vs0/bare -n ns2-root -L1G
# lvcreate /dev/vs0 -n ns2-swap -L1G
# mkswap /dev/vs0/ns1-swap

With a beginning image for my nameserver, I then had to set up Xen to know how to start the image:

cat > /etc/xen/auto/ns2 << EOF
kernel = "/boot/vmlinuz-2.6-xenU"
memory = 64
name = "ns2"
nics = 1
ip = ""
gateway = ""
disk = ['file:/dev/fs0/ns2-root,sda1,w']
root = "/dev/sda1 ro"

My virtual server was almost ready. I just had to tune a couple things on the ns2 image to make it ready for boot

# mount /dev/fs0-nfs2 /mnt
# [edit /etc/hostname to reflect]
# umount /mnt

Finally, time to start up the image and make it do something useful:

# xm create -c /etc/xen/auto/ns2
[watch the system boot up]

Great! The system is up. It just so happens that the root account has an ssh key from my laptop, so I ssh in..

# ssh
# apt-get install bind
[ here I also scp'ed in the zonefiles, etc, restarted
bind, etc. Just the same as if it were a completely normal system]

I have one note about the networking: Xen0 bridges connections for the virtual machines. This means that if you set Xen0 up for and put the virtual machine on, then you won't be able to ssh to the machine because they're on different networks. Xen seems to have a problem with routing when ip aliasing is involved.

All said and done, it took me about 8 hours to get this working for the first time. Now that I have things set up correctly, I can make a new server in about 20 minutes.

And that's really basically it. I repeated it a few times for a handful of other machines. Each time I grew a bigger smile on my face as I considered how how much my electric bills are going down.

Why I Contribute to Free Software

One of my interests includes writing software. By writing software, I get to create things that are used from anywhere from myself up to literally millions of people. Thus, by putting my efforts into free software, I can do my own little part to help everyone else.

The particular type of software that I write is called "free software". Free software is free not just in cost, but in other ways as well. For example, others can take parts of my hard work and use this work in their work.

One of the most attractive things about free software is that my contributions to society don't get used up. The more people that use my work, the more its done for others. Is there any other way to give that keeps giving?

Gnome Conf

Noticed that one of my compatriots, Jeff Waugh (aka jdub) is heading on a cross country US tour to places like the Gnome Developers conference. On this tour he'll primarily focus upon Ubuntu. I wouldn't be suprised to hear that he talked about Bazaar-NG more than a couple times.

Those of you that have the priviledge to meet Jeff. Jeff is larger than life. He's eloquent, intelligent and, perhaps most importantly, hilarious!

I'm considering heading up to the Gnome developer conference for the purposes of studying his public speaking kung-fu. :)