The mutterings of a programmer

Moving my blog

2005-12-30T20:18:00.000-08:00

I've moved my blog to jblack.linuxguru.net

Sorry for the inconvienance.

Ubuntu Below Zero Photos

2005-12-03T20:17:00.000-08:00

I've put up my Ubuntu Below Zero pictures. These pictures manage to catch the last half of UBZ. Many of these photos were captured in darker areas; some with flash and some without.

I have also kept these photos in their original, untouched format. Please feel free to ask for help if you don't know how to remove red eye from photos.

Onto the photos! Ubuntu Below Zero photos

Test Driven Development

2005-11-30T22:44:00.000-08:00

I've spent the last week or so pairing with Robert Collins (Bazaar-NG, Squid). His solitary goal was to force feed me TDD whether I liked it or not. He wasn't interested in my concerns about whether or not TDD caused problems with design. He was deaf to my worries that the code we worked on wouldn't result in solid testing. Nope. Not Rob. "I'm not going to make you do this for the rest of your life, but you're going to do it this week."

I jumped in with the promise that this particular tunnel really did have an end. The worst that could happen was that I would have to load up on tests and rewrite the code so that it would be sufficiently flexible to evolve over its lifetime. The best that could happen is that I would have to swallow my pride and say that he was right.

I didn't start writing test cases for my code until about a year ago. There wasn't much point; anything that I could write a test case for, I could do more flexibly by hand. I just didn't like the idea of putting aside real code for the purpose of fake code with the sole purpose of making the real stuff do some functional equivilant of jumping jacks.

Somebody once pushed me to try using them anyways. He suggested. He prodded. He reminded. He pushed. He said something very similiar to what Rob did: "Do it for a week and then decide whether or not there's use".

So I tried it. When I found a bug in a program, I wrote a test case. If I wrote code that was too complicated to test by hand, I wrote a test case. I even write test cases when submitting code to someone else.

Test cases are more like jumping jacks than I realized. Each test case is like hopping for a jumping jack - a little bit of effort and no real movement. But when does a lot of jumping jacks, something magical happens. Bodies strengthen. Health improves. Sickness is fought off more easily. Lots of good things come from jumping jacks.

The exact same thing happens with test cases. Your codebase, with a lot of good test cases, becomes strong. A small change over here won't cause subtle breakage somewhere else. If it does, your test cases will scream -- loudly and clearly -- that something has gone terribly, terribly wrong.

Test Driven development is more of the same thing with one notable exception: Write test cases first and code second. After all, isn't it better to do a few jumping jacks at a time when you're still young and healthy... or after you've already gotten fat and lazy?

Oh, I almost forgot to mention. The same person that turned me onto Test Driven Development is the same Rob Collins that got me into testing code in the first place.

Thanks Rob.

Blog me Bentley

2005-10-02T15:36:00.000-07:00

I just read Aaron Bentley's blog about Larry McVoy's latest infamous stunt.. McVoy seems to continue in the same nefarious activities as he has done before; namely, to threaten any company using bitkeeper that hires developers of free software revision control systems.

Whether or not McVoy can enforce these claims is only a question that the judicial system can answer. Regardless, Its. Just. Plain. Evil. Imagine if other companies behaved in this manner: McDonalds firing suppliers that ate at Taco Bell; American Airlines refusing to fly pilots of other airlines... the examples could go on.

The Doldrums of History

2005-09-18T06:26:00.000-07:00

In a recent paper I wrote that a revision control system with lossy storage is a possible way to get around the problems that typically affect revision control. The reactions to this were not very surprising, typically reducing to something along the lines of 'Dude, storying all of the past storage is the whole point to revision control!'

Essentially, the RCS field has come to terms with the thought that storing the full history is a basic truth of revision control system. The reasoning that leads to this position is generally pretty pretty good. Several lines of reasoning support this assessment with three of the most important ones being: merging capabilities are affected by the amount of history present, the person responsible for the codebase may come under legal scrutiny current and users have the expectation that anything they store they will be able to retrieve.

This evidence, though it appears to be solid on the surface, is actually circumstantial. In this paper I'll examine the three reasons at length and illustrate that storing full history, while potentially useful, is not actually an inherent truth of revision control systems.

The first supporting reason for lossless revision storage is that merging is affected by the history present. This statement is certainly true with certain implementations; a user attempting to perform a merge with history related to the merge with the RCS system named "tla" is in for a hard row indeed. Other revision control systems, such as Bazaar, are able to partially sidestep the problem by searching for, and usually finding a suboptimal, but fully traceable, history. These two tools have this obligation because they rely upon something called three-way merging. The details behind three way merging are complex, though a oversimplified explanation will suffice for this conversation: A three way merge figures out which patches to apply by finding a previous point in time in which the two branches were identical. The changes between the common ancestsor and the banch to be merged are figured out and then these changes are applied to the other branch.

Other revision control systems, such as bazaar-ng, are seriously discussing avoiding three way merges entirely. The process is a bit different, but presumably also requires the knowledge of what modifications have already been applied. By applying just the patches that are missing, one can perform the equivilant of three way merge without actually performing a merge. This means that finding a common ancestor is no longer obligatory.

The full history is not required to perform these merges. The only fundamental component that is strictly required to perform a three way merge is a record of what has been previously applied so as to avoid applying it again. This does not mean that many revision control systems do not keep the full history (many of them, such as Bazaar, do so by default). This extra information isn't kept strictly to perform a three way merge; rather its kept in order to perform more advanced types of three way merges such as intentionally storing a false copy of accurate history (which I'll henceforth refer to as "a corrupted merge"). In simpler terms, *a record of what history has been seen is required, but not the history itself*

Prior to going into this argument I should clearly point out that I do not qualify as a lawyer, a paralegal or for that matter a filing clerk at the courthouse. The information contained herein is not intended to be legal advice. If you are seeking legal advice then I suggest you seek a lawyer for lawyerly things.

The second listed reason for indefinitely history is that the person maintaining a branch or archive may come under legal scrutiny. A compelling example of this reasoning is a court case (still going on at the time of this writing) named SCO Vs. IBM. Though SCO's arguments are rather unclear, the cental tenant of their argument seems to be that IBM misapropriated source code that belonged to SCO was unlawfully given to the Linux kernel. The Linux community reacted quickly to these charges by scouring the source code for submissions that were potentially infringing. The result of this search is generally considered negative; the public is not able to identify infringing code. This case is very well known, so for the sake of discussion lets assume that infringing code had been found. We'll also assume, again for the sake of argument, that IBM was the proper entity to litigate against.

The assumption that infinging code had beenperson that found makes no difference, as legal responsibility flows upwards. The person ultimately legally responsible for an infringing action is not the person that performed the infringement, but the highest ranking executive (or lead developer for those projects that do not have a formal organization) that had control over the organization's policies. The failure, though created by a typically low ranking individual, should have been caught misapropriating code by policy and workflow monitoring that ensured that such infringement could not happen without being caught eventually. This can be illustrated by examing the policies that Linus Torvalds put into place after the initial SCO allegations; he would be even more guilty than IBM for having first accepted (via delegation), then propogated the infringement. To avoid this sort of repeat Torvalds now requires that each person involved in the chain of merging of a submission sign off on code that it is not infringing.

This clearly illustrates that clearly identifying exactly who contributed what code at what time becomes less important as the longer the infringing code remains in the codebase, the more responsible the maintainer is and the less responsible the submitter. Regardless, its typically
nice to be able to point the finger at somebody else and blame someone else on a moral level, even if legal responsibility can not be evaded. This feel-good act is actually ultimately self-destructive.

The majority of programmers live under legal systems that are very closely related to the British style of judicial reasoning. In these legal systems plausible deniability is a an appropriate legal excuse. Any entity being sued another one looses the ability to destroy any of the entity's history or risk charges of destruction of evidence, tampering with evidence, interfering with an investigation and so on. Organizations will typically evade this sort of problem by intentionally intituting policies that ensure for the destruction of all possible company records after the elapsation of any legally mandated minimums.

These two facts put side by side show that from a legal position indefinitely storing history is not a net win; rather its a drawback. Storing as much history as possible will not provide a legal defense, but will hinder one.

The final listed reason for storing full history is that the user of a revision control system typically expects to be able to pull out by demand the representation of his software at any arbitrary point in the past. Supposedly any revision control system that is not able to perform this action is not a revision control system at all, which can pose interesting questions for whether systems like git is a revision control system at all.

This expectation does not meet with actual practice. The value of any given revision of software is inversely proportional to the distance from the current version. As a case in point, consider which version of his software a developer would rather loose: a version from two weeks ago or a version from two years ago. Additionally, the further back in time a developer looks, the less granularity that a user needs. A user may wish to distinguish the code from today from the code from last thursday, or the code from the beginning of this month from the beginning of two months ago, or today's code from two years ago. However, when users are comparing today's code against code from two years ago aren't generally doesn't care if the the version from two years ago happened on a Tuesday or a Wednesday.

Unfortunately most contemporary revision control systems treat all versions equally. The baggage that comes with storing the third revision alongside the six thousandth revision is for all practical purposes equivilant. Thus, a version of inherently less value costs just as much as current software.

The equal cost for revisions means that as the number of revisions increase, the cost for working with the newer revisions will increase. In many RCSs the definition of the current version is in part defined by the sum of its previous versions with a little bit more added. Different RCSs handle the past history differently; Arch and derivitaves such as Bazaar store this previous history as changesets. Others, such as Bazaar-NG, store the changes on a per-file basis. Regardless, typically all previous revisions are there in some form or another, standing by just in case they are needed by the user or the RCS.

Many revision control systems,including Bzr for the not-most-current case, use a recursive algorithm: To get a particular version of a file or changeset, one applies a set of changes to the previous file or revision, which in turn will apply a set of changes to the previous file or revision. This recursive algorithm is the achilles heel of revision control systems.. The achilles heel shows itself in two ways. The first problem is that in order to use the latest versions one must carry around the older revisions as well. The second problem is that in order to access the latest versions one must access the older versions.

The result of these two problems is that the cost of any current version is the sum of the cost of all previous versions plus a little bit extra. Thusly, as versions are saved, the cost goes up while the worth declines. Remember: any given version has less value as it ages. The resulting inflation, because of the recursive nature, affects the latest versions most strongly.

Many people have been seen stating that a good revision control system solution is similiar to a good backup solution. Surely this premise applies to a revision control system then it would apply to a backup system as well. If these two types of systems are similiar, then why doesn't one see this sort of lossy approach taken with backup systems? The simplest answer is that one does see *exactly this sort of behavior* in some backup systems. The traditional unix tape rotation system is explicitely designed to provide long term storage with a level of looser granularity as you look at older tapes. One will be able to find a tape for a specific day in the last two weeks, a tape for a specific previous week for the last month, a specific month in the last year and so on. Rarely does one find a backup system run by a highly qualified admin that stores a daily version going back to the project's inception (or even daily for the last years). Additionally, many moden revision control systems handle this rotation automatically by systematically pruning data that closely resides next to other older stored data.

Maintaining all previous history is not required. The common reasons for maintaining all history actually provides stronger arguments for tracking history and not maintaining it. This leaves RCS developers with a much more interesting question: When should things be tracked and when should things be maintained?

The Achilles heel of DRCS

2005-09-13T19:37:00.000-07:00

One of the things that most distributed revision control systems do is that they keep a permanant, indefinite record of everything that's ever happened to a branch. This can be useful if some company in Santa Cruz decides to defame you or your company. Most of the time this older information is unnecessary baggage that is essentially kept around just in case its ever needed.

This is the achilles heel of distributed revision control. As time passes the storage of past events grow indefinitely and become unwieldy to work with. Some tools, such as bazaar-NG, are working to address certain aspects of this.

Whats needed is a way to conflate history by merging together old revisions. By conflating revisions a bit of disk space is saved, network access is reduced and the amount of work involved in building by using ancestors is reduced.

Consider, for example, some sort of pool of patches. If the access to these patches is monitored then one can detect which patches are no longer very interesting. If two non-interesting patches are next to each other, then merge them into a single superpatch.The two patches in the pool can then be removed.

As time passes older, unused patches will continue to clump together. These multipatches can be used in the place of the individual patches in any place that all of them would have been applied anyways.

If the user wants a revision that has been merged into a multipatch, just go ahead and get it and add it to the pool. It can be munged with other patches just as with prior events.

Bazaar-NG Hackfest coming up

2005-09-13T05:32:00.000-07:00

A bazaar-NG hackfest is coming up this November. The event is going to be in Toronto, Canada and will be work, work, work. If you like to work your ass off for free and make cool things happen, then I'll meet you there. :)

If you would like more information about the hackfest, then you can read about it at The Ubuntu Below Zero Page.

New article

2005-09-11T19:19:00.000-07:00

Tonight I wrote a new article for linuxgazette.com. This article covers why distributed revision control is a good thing. You can find the article at My first LinuxGazette article

My first impressions of full ubuntu

2005-09-10T20:48:00.000-07:00

I have been using ubuntu since before it was called ubuntu (back then it was no-name-yet). I was also the first person that used KDE in the Ubuntu world, predating the efforts of noble projects such as Kubuntu.

I did this not because I happen to be a great KDE fan, but because the gnome software wasn't capable of properly handling my dual head setup properly. If you've ever run a dual-head setup yourself (this is not only possible but reasonably easy with most laptops), then you'll understand the importance of both maximize only applying to one monitor and having windows be created on the same screen that the mouse pointer happens to be on.

The ACPI team has informed me that they plan on hijacking my laptop, a behemoth Sony Vaio A190, while meeting in Canada. This gave me a great excuse to get another laptop. This new one, a T350P is as small and long lasting as the A190 is large but short lived.

Ironically, the new T350P is even more odd than the A190 This machine runs Ubuntu for approximately six hours on a single charge, has a widescreen 1280x768 display and gobs of ACPI problems, most of which have been fixed in the official Ubuntu.

So far my opinion of gnome on the T350P is rather mixed. Watching the processor speed is almost as nice as being able to control the lcd brightness. With the old laptop I was able to apt-get install just about anything I wanted and have the software show up in K->Debian menu. On the little laptop I haven't figured out how to get unofficial software such as pingus to show up in the gnome menus.

Gnuarch firestorm pt. 2

2005-09-07T21:31:00.000-07:00

As the gnuarch firestorm continues on... well, you read the link and figure it out. :)

Secretaries sacked after cyber brawl | NSW/ACT | Breaking News 24/7 - NEWS.com.au (08-09-2005): "'She's a very nice, quiet girl,' Mr Bird said. 'It is just a silly girl thing that got out of control.'"

Individuals in a Capitalistic Society

2005-09-07T11:48:00.000-07:00

The gnuarch mailing list is under fire again. Companies are evil this, venture capitalists are slash and burners that.. the same old story from you know who. When I hear tirades such as these, I can't help but consider the motivations of the writer. Is the person complaining that the system does not work at all, or is the individual is complaining that the system does not work for them?

The prototypical industrialized society is framed around the capitalist system. This system provides for the private ownership, either individually or as groups, to conduct commerce interdependantly with a minimal amount of government controls. An entity can focus on the things that it is good at and trade resources (typically capital, workprodcut, goods or ingenuity) for the things that it is not. This specialization allows for much greater efficiencies than any person could achieve individually.

A player in the capitalist model, whether its an inventor, an investor, a small company or a large corporation, is essentially a living organism that requires certain resources to exist. These players, instead of living off of salads and cheeseburgers, live on capital. Any entity in the capitalist model that runs out of capital will starve and die no less surely than any breathing creature that runs out of food.

This means that the capitalist system is, to some extent, darwinistic. Players, in order to successfully compete against other players, will maximize their value (price vs. worth) by making arrangements with the most effciencient players that they have access to. Failure to do so results in less capital as the competiting players become more efficient than the players that don't.

The price of a product is not limited to the initial cost. Direct factors include, but are not limited to: transportation costs, resale value, maintenance costs. Indirect price factors include the cost of maintaining the relationship with the vendor such as, but again not limited to: Effort contacting the vendor, the integrity of the vendor and the cost of maintaining a positive relationship with the vendor.

A key point that many individuals fail to realize is that individuals are mandatory participants. An individual, just the same of a large multi-regional conglomerate, has a certain amount of resources, whether its capital (venture capitalists), workproduct, ingenuity or other things. These resources can be traded for other resources such as a paycheck, partial company ownership, health care, pensions and so forth. These resources are used to keep the indivitual playing in the game, so that they are able to earn (and potentially reinvest) for more capital.

Just as with any corporation, an individual can price themselves out of the market in any of a variety of ways. One of the many ways that an individual can undercompete is asking for an amount of capital that exceeds thier worth as perceived by others. Another, more subtle, way to undercompete is to make the cost in effort of working with the individual high enough that potential worth is overwhelmed. An individual that is unreliable, inconsistant and works poorly with others may not be able to trade his inherent worth at any price (even free).

Thus, any person or company that finds itself running out of capital must immediately improve the offered value of its goods or services or face failure. Failure to do so can result in metaphorical (literal in the case of the individual) starvation. Thusly, a player must reduce either cost or increase worth. Increasing worth over the immediate term is typically not practical. Reducing cost, however is.

For an individual, one can either reduce the price for doing business or reduce the cost of doing business with the individual. An individual faced with the need to immeidately reduce cost has options available to him or her, including: Improving reliability, improving consistancy, improving honesty, improving relations with others and getting along better with others.

Fighting with others, complaining and other non-constructive acts while under the threat of starvation is a short sighted methodology that will not work.

Tracking too many irc channels

2005-09-05T17:16:00.000-07:00

As part of my job I have to track a lot of irc channels. Throw on top of the ones that I'm personally interested in, I have to watch a lot of stuff. Theres far too many channels to constantly switch from window to window; one would literally wear out the /win keys on the keyboard. So how does one monitor multiple channels at once?

Simple, really:

Turn on logging in irc (I believe that's /log *)
Setup a tall console with the smallest font that you can easily read
Tail -f /var/log/freenode/*

And thats all there is to it. Now you can easily keep an eye on dozens of conversations at the same time. :)

Live archive support

2005-09-05T08:27:00.000-07:00

As many of you are aware, recently the old sourcecontrol.net server suffered massive filesystem corruption (thanks Hans). The machine is back up and configured and is all ext3. The new machine, that used to be called push.sourcecontrol.net, is now codehome.revisioncontrol.net.

Reaching the machine via ssh/sftp is a little tricker than before. One can do this by adding the following to their ~/.ssh/config:

Host codehome
Hostname revisioncontrol.net
Port 1008
ProxyCommand none

Reaching the machine via http is a great deal easier. In that case, one simply points their browser or bazaar at:
http://codehome.revisioncontrol.net/~username/. Listing files will not be necessary as I intend to support readable webdav.

As with before this machine lives on my connection at home. Thusly, if its reasonably possible, use mirrors.sourcecontrol.net. If for some reason mirrors.sourcecontrol.net won't work for you, get ahold of me and I'll set you up with an account.

For those that it will help, I have a backup of the system from October of 2004, which was the date that the supermirror officially moved to Canonical.

Bazaar-NG Wiki updates

2005-09-05T08:16:00.000-07:00

Good documentation is important for any project that wants to grow up strong and healthy. Unfortunately, doing so is just about as tasty as eating vegetables.

I've just finished updating the bzr wiki site a bit. I pulled several things off the mailing list, updated various version numbers, and rewrote the Bzr page.

Getting community stuff going

2005-09-05T03:24:00.000-07:00

Now that I finally have a good handle on Xen I'm starting to do some excellent stuff with it. In the space of about five minutes I can get a new host up on the internet.

I had gotten rather cramped hosting the stuff that I was. I'm very happy about being able to make virtual machines that are exactly tuned to one specific thing. Now I don't have to worry about which people have access to which things via which methods. I just make a new machine for each thing. :)

Tonight I also setup a planet for revision control users and developers. You can find it at http://planet.revisioncontrol.net. I also did a bit of tuning at the bazaar.canonical.com wiki and setup a revisioncontrol.net wiki which is essentially empty.

If you work on a free software revision control system and have a blog, then I'm looking for you! Shoot me a mail (jblack@gnuarch.org will work as well as anything) and I'll make sure to get it added to the planet.

I also plan on setting up rss feeds for other stuff as well, so that I can planetize them as well.

Poor english

2005-09-04T06:43:00.000-07:00

After screwing with xen for the last four hours I decided to wind down with some netstumbler. One of the pages that I came across was: Linux Shell Scripting Tutorial - A Beginner's handbook. As the title states, this is a beginner's book. The quality of the information is actually pretty good once you get past the poor english within the document.

Its people like Vivek that warm my 'hart' on cold winter days. I see so many people that couldn't care less for others. Vivek, despite his english limitations, is clearly going through great personal effort to help others. This effort is admirable. Vivek, your first beer in Pennsylvania is on me.

In closing, I include some poetry from our writer:

I love linux.

It is different from all other Os
My brother Vikrant also loves linux.
He currently lerarns linux.
Linux is cooool.
Linux is now 10 years old.
Next year linux will be 11 year old.
Rani my sister never uses Linux
She only loves to play games and nothing else.
Do you know?
(DOT) is special command of linux.
Okay! I will stop.

Notes From Inside New Orleans by Jordan Flaherty

2005-09-03T21:44:00.000-07:00

Notes From Inside New Orleans by Jordan Flaherty
Friday, September 2, 2005

Thanks to all the loved ones and long-lost friends for your sweet notes of concern, offers of housing and support, etc. Yes, I stayed through the storm and aftermath. I'm fine - much better off than most of my brother and sister hurricane survivors. Below is my attempt to relay some of what I've seen these last few days.

I just left New Orleans a couple hours ago. I traveled from the apartment I was staying in by boat to a helicopter to a refugee camp. If anyone wants to examine the attitude of federal and state officials towards the victims of hurricane Katrina, I advise you to visit one of the refugee camps.

In the refugee camp I just left, on the I-10 freeway near Causeway, thousands of people (at least 90% black and poor) stood and squatted in mud and trash behind metal barricades, under an unforgiving sun, with heavily armed soldiers standing guard over them. When a bus would come through, it would stop at a random spot, state police would open a gap in one of the barricades, and people would rush for the bus, with no information given about where the bus was going. Once inside (we were told) evacuees would be told where the bus was taking them - Baton Rouge,
Houston, Arkansas, Dallas, or other locations. I was told that if you
boarded a bus bound for Arkansas (for example), even people with family and a place to stay in Baton Rouge would not be allowed to get out of the bus as it passed through Baton Rouge. You had no choice but to go to the shelter in Arkansas. If you had people willing to come to New Orleans to pick you up, they could not come within 17 miles of the camp.

I traveled throughout the camp and spoke to Red Cross workers, Salvation Army workers, National Guard, and state police, and although they were friendly, no one could give me any details on when buses would arrive, how many, where they would go to, or any other information. I spoke to the several teams of journalists nearby, and asked if any of them had been able to get any information from any federal or state officials on any of these questions, and all of them, from Australian tv to local Fox
affiliates complained of an unorganized, non-communicative, mess. One cameraman told me "as someone who's been here in this camp for two days, the only information I can give you is this: get out by nightfall. You don't want to be here at night."

There was also no visible attempt by any of those running the camp to set up any sort of transparent and consistent system, for instance a line to get on buses, a way to register contact information or find family members, special needs services for children and infirm, phone services, treatment for possible disease exposure, nor even a single trash can.

To understand this tragedy, its important to look at New Orleans itself.

For those who have not lived in New Orleans, you have missed a incredible, glorious, vital, city. A place with a culture and energy
unlike anywhere else in the world. A 70% African-American city where resistance to white supremecy has supported a generous, subversive and unique culture of vivid beauty. From jazz, blues and hiphop, to secondlines, Mardi Gras Indians, Parades, Beads, Jazz Funerals, and red beans and rice on Monday nights, New Orleans is a place of art and music and dance and sexuality and liberation unlike anywhere else in the world.

It is a city of kindness and hospitality, where walking down the block
can take two hours because you stop and talk to someone on every porch, and where a community pulls together when someone is in need. It is a city of extended families and social networks filling the gaps left by city, state and federal goverments that have abdicated their responsibilty for the public welfare. It is a city where someone you walk past on the street not only asks how you are, they wait for an answer.

It is also a city of exploitation and segregation and fear. The city of
New Orleans has a population of just over 500,000 and was expecting 300 murders this year, most of them centered on just a few, overwhelmingly black, neighborhoods. Police have been quoted as saying that they don't need to search out the perpetrators, because usually a few days after a shooting, the attacker is shot in revenge.

There is an atmosphere of intense hostility and distrust between much of Black New Orleans and the N.O. Police Department. In recent months, officers have been accused of everything from drug running to corruption to theft. In seperate incidents, two New Orleans police officers were recently charged with rape (while in uniform), and there have been several high profile police killings of unarmed youth, including the murder of Jenard Thomas, which has inspired ongoing weekly protests for several months.

The city has a 40% illiteracy rate, and over 50% of black ninth graders will not graduate in four years. Louisiana spends on average $4,724 per child's education and ranks 48th in the country for lowest teacher salaries. The equivalent of more than two classrooms of young people drop out of Louisiana schools every day and about 50,000 students are absent from school on any given day. Far too many young black men from New Orleans end up enslaved in Angola Prison, a former slave plantation where inmates still do manual farm labor, and over 90% of inmates eventually die in the prison. It is a city where industry has left, and most remaining jobs are are low-paying, transient, insecure jobs in the service economy.

Race has always been the undercurrent of Louisiana politics. This
disaster is one that was constructed out of racism, neglect and
incompetence. Hurricane Katrina was the inevitable spark igniting the gasoline of cruelty and corruption. From the neighborhoods left most at risk, to the treatment of the refugees to the the media portayal of the victims, this disaster is shaped by race.

Louisiana politics is famously corrupt, but with the tragedies of this week our political leaders have defined a new level of incompetence. As hurricane Katrina approached, our Governor urged us to "Pray the hurricane down" to a level two. Trapped in a building two days after the hurricane, we tuned our battery-operated radio into local radio and tv stations, hoping for vital news, and were told that our governor had called for a day of prayer. As rumors and panic began to rule, they was no source of solid dependable information. Tuesday night, politicians and reporters said the water level would rise another 12 feet - instead
it stabilized. Rumors spread like wildfire, and the politicians and
media only made it worse.

While the rich escaped New Orleans, those with nowhere to go and no way to get there were left behind. Adding salt to the wound, the local and national media have spent the last week demonizing those left behind. As someone that loves New Orleans and the people in it, this is the part of this tragedy that hurts me the most, and it hurts me deeply.

No sane person should classify someone who takes food from indefinitely closed stores in a desperate, starving city as a "looter", but thats just what the media did over and over again. Sherrifs and politicians talked of having troops protect stores instead of perform rescue operations.

Images of New Orleans?hurricane-ravaged population were transformed into black, out-of-control, criminals. As if taking a stereo from a store that will clearly be insured against loss is a greater crime than the governmental neglect and incompetence that did billions of dollars of damage and destroyed a city. This media focus is a tactic, just as the eighties focus on "welfare queens" and "super-predators" obscured the simultaneous and much larger crimes of the Savings and Loan scams and mass layoffs, the hyper-exploited people of New Orleans are being used as a scapegoat to cover up much larger crimes.

City, state and national politicians are the real criminals here. Since
at least the mid-1800s, its been widely known the danger faced by
flooding to New Orleans. The flood of 1927, which, like this week's
events, was more about politics and racism than any kind of natural
disaster, illustrated exactly the danger faced. Yet government
officials have consistently refused to spend the money to protect this poor, overwhelmingly black, city. While FEMA and others warned of the urgent impending danger to New Orleans and put forward proposals for funding to reinforce and protect the city, the Bush administration, in every year since 2001, has cut or refused to fund New Orleans flood control, and ignored scientists warnings of increased hurricanes as a result of global warming. And, as the dangers rose with the floodlines, the lack of coordinated response dramatized vividly the callous disregard of our elected leaders.

The aftermath from the 1927 flood helped shape the elections of both a US President and a Governor, and ushered in the southern populist politics of Huey Long.

In the coming months, billions of dollars will likely flood into New
Orleans. This money can either be spent to usher in a "New Deal"
for the city, with public investment, creation of stable union jobs, new schools, cultural programs and housing restoration, or the city can be rebuilt and revitalized? to a shell of its former self, with newer hotels, more casinos, and with chain stores and theme parks replacing the former neighborhoods, cultural centers and corner jazz clubs.

Long before Katrina, New Orleans was hit by a hurricane of poverty, racism, disinvestment, de-industrialization and corruption. Simply the damage from this pre-Katrina hurricane will take billions to repair.

Now that the money is flowing in, and the world's eyes are focused on Katrina, its vital that progressive-minded people take this opportunity to fight for a rebuilding with justice. New Orleans is a special place, and we need to fight for its rebirth.

What's Wrong With Sorcerer?

2005-09-03T01:18:00.000-07:00

Here's a post from a friend of mine, Dennis. I suspect that he's a bit pissed off about people changing interfaces too quickly. Does anybody else know someone like this? I Hate Blogs: What's Wrong With Sorcerer?

A quote from New Orleans Mayor

2005-09-03T01:05:00.000-07:00

Ray Nagin, mayor of New Orleans, had this to say about a week after the great New Orleans flood:

Don't tell me 40,000 people are coming here. They're not here. Its too doggone late. Now get off your *** and lets do something and lets fix the biggest *** crisis in the history of this country

He has a point. As Americans throughout the country mobilize, gather food and water, and send support down to Louisiana one can't help but wonder why its not already there. I have no doubt that if Hollywood had burned down, the cargo jets would have flown, the trucks would have gone rolling and they would have omlettes for breakfast.

Most Recently Eligible Bachelor

2005-09-03T00:00:00.000-07:00

I received a call from my lawyer today. I found out that my divorce is finally final. What a mess. Some people say "don't get divorced". I say "don't get married".

On a more positive note, my ex-wife told me that they were collecting water at the local mall for the Louisiana tragedy. Some of those people out there are dying for lack of something to drink. With this thought in mind, I accepted the ex's offer to use her van to carry water.

Unfortunately, the nearby store was nearly out of water. They're entire stock of gallon jugs of water were gone. Fortunately, not many people realize that distilled water is perfectly drinkable. Well, that was until I their last 90 gallons. Now they really are out of water. This tragedy makes the 2005 Wilkes-Barre, PA flood look like the Sahara.

On a more trivial note, I noticed that the Better Business Bureau hides the Microsoft complaints(normally they list how many complaints were made, how many were resolved, how many weren't). I wonder why that is?

Your own mainframe!

2005-09-02T19:15:00.000-07:00

Yesterday I worked with Xen for the first time. Only two words truly apply: holy wow!.

For those that haven't seen it before, Xen is a suit of programs and kernel hacks that give you the ability to run multiple installations on one machine simultaneously. In simpler words: one can have the logical equivilant of a mainframe on commodity computers.

This ability is supposedly similiar to other technologies such as UserMode Linux, VMWare and Bochs. I've never used these due to various other problems. I found UML too difficult to get working on Ubuntu. Bochs is supposedly full emulation and runs slowly. VMWare not only supposedly runs poorly as well, but is proprietary to boot. These problems no longer matter to me as I've completely fallin in love with Xen.

This post is about how I got Xen running. Getting to this point involved taking a few wrong turns. If you go about it this way, you shouldn't have nearly the problems that I did.

The first step is to get your hands on a machine that is not too old. I personally would suggest nothing less than a post-2003 machine with no less than 512 megs of ram, 2Ghz processor and an empty 60 gig hard drive. Older machines may work, but my first attempt on a 700Mhz machine with 256 megs of ram failed miserably as the machine's old ACPI setup did not work well with Xen.

I also highly recommend using LVM, as you're going to want to create and delete a lot of partitions without bringing down the machine. Though the Xen docs suggest using 64 megs of memory for the Dom0 superserver, I recommend 256 megs of memory. Otherwise, the system is likely to crash on you when you do any sort of significant LVM operation.

The next step is to install a bare bones linux distro on the machine. You can use whatever you want here. I chose Ubuntu. While installing ubuntu, I set up my partitions thusly:


hda1   ext3  /boot   2 gigs    (room for problems)
hda2   ext3  /       2 gigs (1.5 gigs extra)
hda5   fs0   LVM          50 gigs
hda6   fs0   LVM          50 gigs
hda7   fs0 LVM          50 gigs
hda8   fs0 LVM          50 gigs

This setup isn't optimal. Unfortunately, I found out the hard way that Xen apparently has problems with root filesystems on LVM. Thusly, I ended up using hda2 for the root filesystem and put swap in the LVM instead. Some day I'll probably resize hda1 and hda2 to make room for swap.

I installed the Ubuntu server with the "server" option. This started with a reasonably stripped down system. I then downloaded the Xen unstable binary package from the xen home page, which installed its own kernel and modules. I then proceeded to strip out just about anything that wasn't required for system boot or basic system administration. I even took out a few things that required me to remove the "ubuntu-minimal" package which normally requires any packages that are normally required.

This left me with a system that worked fine, but was essentually useless. Just to make sure things were working, I downloaded that minimal linux image that Xen has available for download from sourceforge. Sure enough, after editing a config file, I was able to start these images:


# xm create -c ttylinux-xenconf

The system booted up and everything worked fine. Just for fun, I hacked up a half dozen config files and started several more ttylinuxes.

The time came to get serious. I wanted to base my images off of Ubuntu. As it so happened, the root filesystem was a perfect starting point:


# lvcreate -L1G /dev/fs0 -n bare
# mkfs -t ext3 /dev/fs0-bare
# mount /dev/fs0-bare /mnt
# cp -ax / /mnt; cp /dev/* /mnt/dev
# chroot /mnt
# dpkg --purge (all daemons but postfix + ssh-server,
                all modules and kernel stuff, all
                non-ext3 filesystem support, etc...)
# umount /mnt

I now had a nice starting image called "bare". I now use this image whenever I want to add a new system to the machine. Think of it as an /etc/skel on steroids:


# lvcreate -s /dev/vs0/bare -n ns2-root -L1G
# lvcreate /dev/vs0 -n ns2-swap -L1G
# mkswap /dev/vs0/ns1-swap

With a beginning image for my nameserver, I then had to set up Xen to know how to start the image:


cat > /etc/xen/auto/ns2 << EOF
kernel = "/boot/vmlinuz-2.6-xenU"
memory = 64
name = "ns2"
nics = 1
ip = "209.158.45.75"
gateway = "209.158.45.1"
disk = ['file:/dev/fs0/ns2-root,sda1,w']
root = "/dev/sda1 ro"

My virtual server was almost ready. I just had to tune a couple things on the ns2 image to make it ready for boot


# mount /dev/fs0-nfs2 /mnt
# [edit /etc/hostname to reflect ns2.merconline.com]
# umount /mnt

Finally, time to start up the image and make it do something useful:


# xm create -c /etc/xen/auto/ns2
[watch the system boot up]
login:

Great! The system is up. It just so happens that the root account has an ssh key from my laptop, so I ssh in..


# ssh ns2.merconline.com
# apt-get install bind
....
[ here I also scp'ed in the zonefiles, etc, restarted
  bind, etc. Just the same as if it were a completely normal system]

I have one note about the networking: Xen0 bridges connections for the virtual machines. This means that if you set Xen0 up for 192.168.1.1 and put the virtual machine on 192.168.2.2, then you won't be able to ssh to the machine because they're on different networks. Xen seems to have a problem with routing when ip aliasing is involved.

All said and done, it took me about 8 hours to get this working for the first time. Now that I have things set up correctly, I can make a new server in about 20 minutes.

And that's really basically it. I repeated it a few times for a handful of other machines. Each time I grew a bigger smile on my face as I considered how how much my electric bills are going down.

Why I Contribute to Free Software

2005-09-02T14:51:00.000-07:00

One of my interests includes writing software. By writing software, I get to create things that are used from anywhere from myself up to literally millions of people. Thus, by putting my efforts into free software, I can do my own little part to help everyone else.

The particular type of software that I write is called "free software". Free software is free not just in cost, but in other ways as well. For example, others can take parts of my hard work and use this work in their work.

One of the most attractive things about free software is that my contributions to society don't get used up. The more people that use my work, the more its done for others. Is there any other way to give that keeps giving?

Gnome Conf

2005-09-02T14:18:00.000-07:00

Noticed that one of my compatriots, Jeff Waugh (aka jdub) is heading on a cross country US tour to places like the Gnome Developers conference. On this tour he'll primarily focus upon Ubuntu. I wouldn't be suprised to hear that he talked about Bazaar-NG more than a couple times.

Those of you that have the priviledge to meet Jeff. Jeff is larger than life. He's eloquent, intelligent and, perhaps most importantly, hilarious!

I'm considering heading up to the Gnome developer conference for the purposes of studying his public speaking kung-fu. :)

Distributed Revision Control No Threat, Pt. II

2005-08-22T14:17:00.000-07:00

In yesterday's diary I lined out an explanation of why choosing a non-distributed revision control system was a power play. Later I realized that some people would probably say to themselves "Yeah, but I'm not trying to do a power play. I'm just sticking with a tool that I know". Regardless of intent the end result is the same. Any developer that chooses a non-distributed revision control system has also made development more difficult for non-sanctioned developers, which I'll dive into more deeply some other day.

The big scary spider in the minds of many developers is that some day their project might be forked by somebody else. This fear, which is partially justified, presents with a variety of unhealthy behaviors: obfuscated code, poorly documented code, development plans happening in private backroom discussions rather than in public and centralizaton of control are only some of the tactics that some use in order to hold onto a project.

There are at least three scenarios that result in a potential forking situation: temporarily abandoned projects, divergance of interests and poorly managed projects. Each of these driving forces results in a different type of fork that can have different long term results for the project.

A temporarily abandoned project is one in which the developer has gone AWOL. Perhaps he's burned out, holding his coding hostage for pay or overcommitted in his work. For whatever the reason, the lead developer of the otherwise healthy project is no longer actually leading development. Nature, who abhors a vacuum, sees to it that eventually a new lead developer springs out of the ether. The old project lead, realizing that he's no longer in charge, rushes back to the scene and attempts to regain control. In most cases, the new developer steps back and allows the old lead to regain control of the project. In less common cases, especially if the old lead had limited management skills or had shown a previous history of abandonment, the new project lead will continue development against the wishes of the old developer, leading to two projects in the place where one previously existed.

In a non-distributed revision control system, the new lead has to setup a new revision control system, get the code freshly imported, setup accounts.... only then can he actually start pulling in the patches laying around from the various interests. This code is at risk of being lost when the old lead returns. To avoid this loss, the new lead will likely continue development parallel to the old lead, leading to an arms race.

Imagine if the old lead had been using a distributed RCS. The new lead would have branched from the old lead and continued development. The returning old lead would simply be able to merge the new lead's code and development would continue as if nothing had ever happened. No work is lost, no pride is at risk, and everyone eventually ends up happy. In the abandonment case a distributed revision control system actually reduces the chance of a long term fork.

In a succesful project the userbase typically grows substantially. Over time the project can realize that it is supporting two or more mutually exclusive goals. A project typically handles this by splitting off a subproject that focuses on the unique subgoals. The long term relationship between these two projects is highly dependant upon the relationship between the two lead developers; in many cases the break-up is mutually agreeable and the two parties work together on the interests that are common.

In a non-distributed RCS setup the divergance of the two projects is encouraged because merging becomes increasingly difficult. Eventually the two projects get tried of passing deltas back and forth and give up working together. In a revision control system the merging of specific patches from two distinct, but related, tree is very simple. This feature encourages more merging between projects and encourages both projects to merge the fixes for the similiar projects between one another. Thusly: Distributed revision control systems minimize divergance between two related projects.

Finally, we come to the case of the mismanaged project. I've often heard the following quote: "The internet routes around damage". In a mismanaged project several members of a project have gotten so fed up a project that they have decided to take on a good deal of additional work just to ensure the project's success. Nobody takes this task on lightly due to the high volume of non-development related work that goes with project leadership. Consider the following effort that typically needs to be expended to start a new/forked project:

A website to gather and disseminate information. The website needs to be well maintained in order to convince users to switch over to the new project.
An IRC channel needs to be manned for both developer discussions and user support. Again, this needs to be done well to encourage mindshare
A bug tracker for the new project is a must for any project of significant size. Be ready to spend anywhere from five to twenty hours a week triaging bugs.
A mailing list needs to be setup and the public informed that it exists.
Third party announcements (release announcements, freshmeat announcements, etc)
Conferences, speaking engagements, trolling user groups, etc may become a requirement.

By the time a group of users is willing to hostily fork a project, the choice of RCS is moot. The user has already undertaken the responsibility for doing a lot of work. The choice to reimport the old project's codebase into a non-distributed RCS vs. the slightly less painful branching command in a distributed RCS is irrelevant. Regardless, the two projects can still take advantage of the good work of the other project by easily merging the good portions while ignoring the bad.

Distributed Revision Control No Threat, Pt. I

2005-08-20T16:18:00.000-07:00

One of the things that Canonical is doing is providing bazaar imports of cvs and subversion archives. These imports (The SuperMirror & Bazaar Imports) give uses a way to try out Bazaar 1.x with projects that already have a long history. Goodies from all walks of life are present: GNU, Gnome, KDE, independant projects. This really gives user a great opportunity to try out Bazaar with existing projects and see how Bazaar meets their needs.

Unfortunately some people are a little worried that these imports give people the power to fork projects. The logic goes something like this: With a distributed revision control system any user can branch a project and not offer merges back into the mainline. Most developers want to protect our software against questionable design, poor workmanship and dangerous thoughts. This goal is an admirable one that most people should subscribe to and one I fully support. However, as with most things, the means one uses in order to justify the ends is just as important as the ends themselves.

Locking people out of revision control is not the cure to ensure the cohesiveness of a project. If you run certain revision control systems (cvs and subversion come to mind) you can dictate which people can and can't use revision control (and in which ways). This barrier to entry is sufficient to discourage some projects from forking into two seperate projects.

Using this sort of tactic is not the right way to ensure the cohesiveness of a project. This power play suffers the same consequences of all power plays: The best politicians win over the best meritocrats. The person that holds the control over the RCS gets to decide who has the ability to develop with the benefits of a RCS. Those that toe the line get favored, those that disagree are held back.

Several years ago Eric Raymond wrote a paper comparing the differences between two styles of development: the cathedral style and the bazaar style. Whether or not one agrees with Raymond's paper, one thing is certainly true: A non-distributed revision control system is the establishment of a cathedral type system. With the power of deciding which people can commit and which people can read archives comes the ability to enforce decisions about which code will be developed when and by whom.

Everybody gets to play in a distributed revision control system. Any person can take the official codebase, study it, learn from it, locally branch it and experiment with it. Each person can develop at their own rate, knowing that they can relatively easily merge in the latest and greatest from the official mainline. When their code is ready to be pulled in and supported by mainline the new developer can send a merge request to the official developers with code that is current with today's codebase changes.

So far we've established that controlling access to revision control is a power play. Some developers are willing to accept this cost to prevent the forking of the community. Tomorrow I'll explain the real factors preventing forking (hint: it has little to do with your choice of revision control system)