I was forwarded a pretty interesting (and funny) email thread on Apache MINA‘s mail list. It seems a MINA developer was upset that a particular checkin did not have @See Javadoc comments. So much so, he vetoed the commit.
“I’m on the project *management* committee.
I would like to see the code you commit adheres to some standard.
Until this is resolved my right to veto holds so please revert your commit until we figure out the best choice for Javadoc. If you won’t revert it, I can do it for you.”
A huge flamewar ensued thereafter. This is why I would never, ever host an open source project at Apache. Its not because a silly flamewar happened over a trivial issue. We had a similar one errupt over Steve Ebersole putting in SL4J in Hibernate. Poor Steve had a few of us flaming him. In true JBossian fashion, Steve, the project lead, basically said, “bleep you. I’ve researched the issue thoroughly and I’ll do what I want and think is right.” No, the actual reason I’d never go to Apache is that anybody is allowed to veto your commit via the Apache rules:
“PS : for the record, here is the veto definition (http://www.apache.org/foundation/glossary.html) :
*Veto* According to the Apache methodology, a change which has been made or proposed may be made moot through the exercise of a veto by a committer to the codebase <http://www.apache.org/foundation/glossary.html#Codebase> in question. If the R-T-C <http://www.apache.org/foundation/glossary.html#ReviewThenCommit> commit policy is in effect, a veto prevents the change from being made. In either the R-T-C or C-T-R <http://www.apache.org/foundation/glossary.html#CommitThenReview> environments, a veto applied to a change that has already been made forces it to be reverted. Vetos may not be overridden nor voted down, and only cease to apply when the committer who issued the veto withdraws it. All vetos /must/ be accompanied by a valid technical justification; a veto without such a justification is invalid. Vetos only apply to code changes; they do not apply to procedural issues such as software releases.”
Imagine going to your boss or project lead at work and saying “I veto this commit.” How hilarious would it be to see the look on their face? Seriously though, democracy just doesn’t work in software development. There are too many ways to skin a cat in software. Its hard enough to get code released on budget and on time without some stupid random schmuck vetoing your work.
BTW, if you think this is funny, check out this old PMC complaint. In this case, it wasn’t a developer vetoing a commit, it was some Apache bureaucrat putting down the hammer.
Bill the Plumber 
May 23, 2008 @ 21:01:07
Emmanuel just has his “important hat” on this week.
FOSS projects in general have issues with ego overstepping productivity and Apache is no exception.
May 23, 2008 @ 21:22:46
Just from my experiences in OSS and closed commercial project, I think a benevolent dictator works much much more efficiently than code by democracy. Apache is just too much the means justifies the end.
May 23, 2008 @ 21:33:21
Bill,
I am so glad you will never develop at Apache.
May 23, 2008 @ 21:37:01
@Maarten:
LMAO! NICE! I had a nice chuckle over that. 🙂
May 24, 2008 @ 04:18:04
I guess their goal is to lose contributors. No wonder the majority of their software is shit.
May 28, 2008 @ 05:07:13
Nothing new though Bill – I don’t think anyone would argue against a benevolent dictator being better than democracy (or in Apache’s case a merit based oligarchy). The problem with benevolent dictators is that they can become selfish dictators. The Apache model is set up to avoid that single point of power without becoming a wasteful democracy.
It’s not as powerful, but more stable in the long run. Why should a community lose their product because a dominant developer has gone off his rocker or vanished from view? Why does Steve get to say “bleep you” above, and why does he get to keep Hibernate if all the developers and users decide they need to leave? Just because he’s the current most active developer? [Well… bad example, presumably Red Hat get to keep Hibernate].
Not that the Apache model removes the chances of selfish dictators. There are examples, but I don’t think this is one of them. It’s also not a simple story of bureaucrats causing trouble – there aren’t as many bureaucrats at Apache as people would like to think (a dozen I would hazard – the board and a few others doing board like jobs). This is the sad story of members of a community increasingly not getting on with each other until something blew.
May 28, 2008 @ 12:16:57
This idea that it is a meritocracy is a farce, especially since any committer can veto your checkin or any member of the PMC can come in and change your commit rules. At Apache, every committer is a dictator, and in this MINA case, Emmanuel was acting like a selfish dictator. Actuall, I view this Apache model more as communism, where everyone is equal no matter how much work they put in. It seems with both the MINA and Geronimo case that the PMC is more interested in creating a utopian society than getting any work done. I have heard similar stories from other Apache committers on different projects as well.
Combining the bureaucratic PMC, one-man-one-veto, and the total loss of brand and control, Apache.org makes it a very unpleasant place to be.
What you don’t understand about the benevolent dictator model is that the dictator still is working with volunteers. If the lead still wants free labor, he’s going to have to appease his minions in one way or another or they will leave. Many times this is a good thing though.
I just don’t see many, if any, selfish dictators combined with a successful open source project, just as I don’t see many one-man-one-veto projects getting beyond the incubation stage.
May 30, 2008 @ 05:40:56
No farce – just different meritocracy bars in each of our descriptions. Any committer/PMC member for that project are there by merit. Once that bar is passed the formal meritocracy ends and you are into the oligarchy and have technical equal say. In reality group management dynamics mean the chair has to give up a bit of that say to be more easily impartial, and the active developers tend to form the core of different opinion groups.
We’re in effect saying the same thing, from the perspectives of different groups. You talk about total loss of brand and control, whereas I view it as not having total loss of brand and control. In your benevolent dictator model comment, the other developers have to leave the project if their dictator goes mental – the dictator keeps brand and control, the other developers lose it. In the consensus model, the other developers retain brand and control and the dictator gets to leave.
Usually a project has the potential space for multiple benevolent dictator roles, so knowing that the loss of a dictator won’t mean brand/control loss is of benefit not only to the non-dictators but to the dictators too.
Company owned Open Source does change this a bit I believe; the dictator only owns the project through sufferance of their employer, so if a dictator goes off the rails the employer can easily replace them with someone else. That helps reduce some of the risk to a community of a benevolent dictator but retains much of the value.
It doesn’t work for community owned Open Source, there the benevolent dictator model is a good startup method, with an underlying consensus model below the leaders being needed as the project grows.
May 30, 2008 @ 18:13:18
No farce – just different meritocracy bars in each of our descriptions. Any committer/PMC member for that project are there by merit. Once that bar is passed the formal meritocracy ends and you are into the oligarchy and have technical equal say. In reality group management dynamics mean the chair has to give up a bit of that say to be more easily impartial, and the active developers tend to form the core of different opinion groups.
You just proved my point. There is no meritocracy, just a bunch of developers working hard to get “tenured”. Apache ended up creating a mediocracy rather than a meritocracy where you have a bunch of individual dictators.
We’re in effect saying the same thing, from the perspectives of different groups. You talk about total loss of brand and control, whereas I view it as not having total loss of brand and control. In your benevolent dictator model comment, the other developers have to leave the project if their dictator goes mental – the dictator keeps brand and control, the other developers lose it. In the consensus model, the other developers retain brand and control and the dictator gets to leave.
I ask, why would I, as a initiator of a project take my codebase and users to Apache.org? I’ve lost all control of the project and brand. There is no incentive for me. Sure, initially, I may get a bump in users because of the weight of the Apache brand, but there is no shortcut to success. Also, in the single-dictator, again, contributors *do* have control, they can just fork the project. Your “consensus model” is an oxymoron as there is no way to obtain consensus with a one-man-one-veto rule with the only option, like the single dictator model, of forking the project.
Also, I’d like to refute this idea that the contributors retain brand rights at Apache.org. For instance, JBoss has been a *huge* contributor to Tomcat for years. If we were unable reach consensus because a specific Tomcat contributor went “mental” and vetoed our commits (or vice versa) and we were forced to fork because of technical differences, we would *NOT* be able to take the Tomcat name with us. Tell me how this is different from the benevolent dictator model?
It doesn’t work for community owned Open Source, there the benevolent dictator model is a good startup method, with an underlying consensus model below the leaders being needed as the project grows.
Doesn’t work? How does Linux, MySQL, Spring, Hibernate, JBoss, Jetty, and Drupal “not work”? I think you have it the wrong way. Where Apache.org might be good to start up the project because of the notoriety of the Apache brand, it doesn’t scale very well once one or more developers unite (Geronimo under IBM) or become too prevalent (Trustin and MINA) and the mediocracy of “I’m on the PMC” and one-man-one-veto is more important than getting something done.
Jul 03, 2008 @ 20:57:44
Sigh, dude get off the Tomcat kick, you don’t own the trademark to Tomcat, you’d be welcome to take it an call it “Craptoad” or whatever you like. Also, stop arguing with Henri about Apache. There are good parts of Apache, there are crappy parts of Apache. Some of the things you bring up are valid, some are not. I grew a little disenchanted with the whole rigamarole over at Apache when I started reading conversations on PMC lists that didn’t need to be private. At the same time there are a bunch of good people participating in Apache projects. To cast them all as a bunch of gloryseeking idiots is to say that everyone in Congress is a crook.
I forgot I commented on Chirino’s blog post in 2006. Oh, and BTW, I tried to use Geronimo the other day and it had the feeling of an abandoned house. Oh look, I probably just pissed off a bunch of people I talk to every day. Crap.
Jul 03, 2008 @ 21:16:25
Tim,
I think you missed my point about Tomcat. My point was that there is no open trademark at Apache. Apache owns the trademarks not you, no matter how hard you’ve worked on the project, no matter if you are the one who brought the IP in or not.
I ask you again: Why would I want to host at Apache? What advantage does it give me? I see nothing but disadvantages and red tape. If you’re a real man you’ll suck it up and build your own brand instead of riding on Apache’s coat-tails.
Finally, sure there *are* great people doing great stuff at Apache, its just sad that because of the Apache rules, random schmucks from the PMC or even one developer can screw it up for these star individuals. I hope episodes like MINA and Geronimo will encourage Apache developers to complain and get things fixed so people like Trustin don’t get fed up and leave. This isn’t just Trustin, I know Remy is ready to quit Tomcat and fork it at JBoss. Andy Oliver wished he never hosted POI at Apache. Just to name a few…
Really, if you want to improve Apache, I think you should take a look at how Bob McWhirter runs Codehaus.org. Now, there’s a site I’d love to host my project at…
P.S. How are you doing BTW? Hope things are good.
Jul 03, 2008 @ 22:08:06
Last question first. Things are good, too busy though. I might just take July 4th off this year. You’ll be happy to hear that I’ve stopped spreading FUD about the GPL.
On Codehaus, sure. In agreement, the philosophy there seems to be, *shrug* everybody get along and just do it. Sometimes it works, sometimes it doesn’t. There are definitely less schoolmarm enforcer types running around demanding reports for “the Board” – in fact, there are no enforcer types because there are no rules except that we all must bow down before Bob. Luckily Bob is both wise, merciful, and generous. I worry at Bob sometimes, what if he wakes up one day in a really bad mood and just decides to destroy us all? Don’t you worry about that? I subscribe to his Twitter feed, the man has a dark side.
In general though, I’m not convinced that open source needs a structure. I have a very bad attitude when it comes to these odd corporate entities we set up to foster communities. They set up artificial hierarchies and power structures that alter the very nature of the core human interactions that are essential the the vibrant volunteer culture that manifests itself as open source. We’ve freed ourselves from the shackles of corporate hierarchy only to reinstitute a new priesthood of false merit upon which we see the same base political struggles! 😐
Crap, didn’t that sound really fancy? See, there I go again with that whole bad attitude thing. I could’ve just said something real about why I fail to participate in these things, but I had to go and make a sarcastic self-referential joke. I guess the bad attitude part of me was turned off by the fact that a lot of these open source foundations attract a set of people who are less interested in coding and more interested in governance to the point where you see the same group of 20 people on some PMC mailing list exchanging 10 page missives about the nature of proper participation.
Yet another reason to avoid Apache.org « Bill the Plumber
Aug 29, 2009 @ 14:28:26