Archive for July, 2010

Why setting up my access point was harder than it had to be

July 26, 2010 2 comments

One reason I always try to be nice to newbies is that there are two different kinds:  people new to everything, and very experienced people who happen to be new to one area.  Everyone was the first kind at one point, and could find themselves being the second kind at any time.

This last weekend I found myself in the second category.  My wireless router died on me again.  I seem to go through about one per year (I suspect inadequate heat dissipation), and was sick of replacing them, so I decided to incorporate its functions into my desktop computer.

I’m about as qualified as you can get for undertaking such a task.  I’ve been using Linux extensively since the late 90’s, and write embedded software for network equipment for a living.  I knew all the pieces needed in generic terms, it was just a matter of finding, installing, and configuring the specific implementations on Ubuntu.

First step:  purchase new hardware.  I needed a second gigabit ethernet NIC for the wired side of the network, and an 802.11g interface for the wireless side.  Complicating matters, since my wife depends so heavily on wireless for her netbook, Nook, and Roku, I wanted to purchase the parts from my local Best Buy instead of my usual online sources so it could be up and running faster.

The gigabit ethernet NIC I didn’t even bother looking up hardware compatibility.  There may be some incompatible ones out there, but I have never seen one.  The 802.11g interface is more complicated.  There is wide support on the client side, but I knew using it as an access point has specific driver requirements.

I did some research on, cross referenced it with the inventory on Best Buy’s website, and came up with one possibility.  Only one?  That threw up a red flag, but Best Buy isn’t exactly known for their stellar Linux support, so I let it go.

I picked it up the next day after work, plugged it in, and it turned out to be v3 of that product, which used a different chipset than I was expecting.  I tried a command I had seen on a few HOWTOs anyway:  iwconfig wlan2 mode master.  Didn’t work.  Made sure I was using the best driver, and did some digging to see if there was a development version I could use, and eventually gave up.

Installing the wired side of the network with DNS, DHCP, NAT, and firewall went smoothly.  At least I knew those pieces would be working when I got the wireless up.

The next day, I returned the wireless interface to Best Buy, did some more research to make absolutely sure I was getting a compatible card, and ordered one online overnight delivery.  Plugged it in the next day, and the “mode master” command still didn’t work.

Now I knew something was up.  After some considerable google-fu, I discovered there is an easy new userspace way of handling all the access point management, but the easiest to find documentation all documented the old, more limited way.  In other words, the “mode master” command doesn’t work on purpose now.  Aside from a short issue with hostapd looking like it starts, but not really because of a setting in /etc/defaults/hostapd, I had it up and running fairly quickly.

So what’s the underlying software engineering principle here?  Outdated documents can sometimes be worse than no documents.  Here is this great new feature that makes it easier to do what I want, but because the docs for it are so obscure, it actually made it harder.

I intend to do more about it than rant on my blog.  Ubuntu will soon be getting some updated documentation courtesy one experienced newbie who just figured it out the hard way.  I consider it my way of “paying” for all the great software I use for free, and encourage all of you who solve similar problems to do the same.

Categories: Documentation, Linux

Why branching is better with DVCS

July 18, 2010 2 comments

Why branching is better with DVCS

A reader pointed out that my article on Why DVCS provides better central control focused more on branching models than on the nature of distributed versus centralized. My response is that models that require more than one or two concurrent active branches have never worked very well with CVCS. That doesn’t mean your company doesn’t have dozens of branches in its CVCS repository, just that only one or two are likely to be under active development at any one time, whereas with DVCS active branches are naturally ubiquitous. In this article, I will explore some of the reasons why.

My remarks have in mind a medium to large-sized commercial development team consisting of several dozen developers or more who all share a certain amount of code with each other. Most of it will also apply to open source development, but most of it won’t really apply if your entire team consists of a only handful of developers.

The most obvious difference is in the merge algorithms, and their attendant data structures. It’s true that DVCS has better merge algorithms, for now. There’s nothing inherent about a centralized model that prevents CVCS from using the distributed merge algorithms, so that advantage won’t last for long. However, there are other factors which I believe inhibit the use of ubiquitous branches with CVCS, even if the merge process is just as clean.

As technically-minded people, we sometimes forget the most important component of any version control system: the human element. The choice of decentralized versus centralized has a large impact on human behavior. I’ll cover four of those impacts: shared resources, deciding when to branch, permissions, and spheres of disruption.

Shared Resources

What do humans naturally do when they share a resource, such as, for example, a central version control server? They form committees and seek to come to a consensus on any decision about that resource. The more people who share it, the worse the “lowest common denominator” decision is. Creative ideas take a long time to come to fruition, because everyone has to be convinced. That’s why there are only one or two active branches of development, because that’s what we could get everyone to agree to.

Contrast that committee approach with a small subteam of 5 or so developers working in the same distributed repository. As long as the interface with the rest of the company remains intact, they are free to try out any idea they think will help move things along faster.

Deciding When to Branch

I often hear CVCS proponents say they are using feature branches effectively, because they create them whenever a series of changes is going to be “long” or “disruptive.” Setting aside the issue for a moment that this decision is most likely made by committee, it’s very difficult to pin down definitions of long and disruptive. I spent most of the last week avoiding checking out because of a series of changes going on that were not originally anticipated to be long and disruptive. With DVCS, you create branches even for the short and simple stuff. If it turns out to be long and disruptive, there’s no change in the process.

A lot of companies create a new branch for each release. As some people are finishing bug fixes, others are ramping up on the next project. We don’t want to branch too early, because then all the bug fixes will have to be merged over, but if we branch too late, we delay the start of the next project. What if the guys who finished their product early can branch at a different time than the guys who are frantically fixing the last minute bugs? Distributed repositories makes this possible, even natural. Also, when every bug fix is a local branch, it’s much easier to merge it into the branches for two different releases.


CVCS lets you set permissions on certain branches, but in practice they are very rarely employed to the granularity they need to be, and tend to be either overly permissive or overly restrictive. DVCS lets you set different permissions on different repositories. In case it isn’t apparent how this is useful, I’ll give you an example that I think is fairly common.

Certain areas of our code are deemed crucial enough that they can only be checked in through a select few gatekeepers. However, the gatekeepers are not always the ones who write the code changes. This last week we had some changes go in that had interdependencies with changes in that crucial code. Trying to get all the pieces checked in through and from different people resulted in staggered uncompilable check ins all week, which is why I was afraid to check out. It would have been much easier if we had a separate repository with open permission on that crucial code to do all that integration work, then get it submitted to a gatekeeper to commit to the official repository in one fell swoop.

Spheres of Disruption

Having more than one repository helps limit the amount of disruption caused by creating and deleting branches. For example, how many active branches do you suppose there are in the Linux kernel development? Dozens or even hundreds probably, but if you go to any given repository, you’ll see only the few that most matter to you. A reluctance to organize and wade through that many branches on a central repository creates a natural tendency to keep the number of branches as low as possible.

When you get down to it, the smallest sphere of disruption is an individual developer. Since I can have my own repository on my desktop, people couldn’t care less about how many branches I keep around for my own purposes. I’ve actually been using a DVCS alongside my company’s official CVCS for that very reason. I make temporary branches all the time to quickly check out that bug a tester just saw, test out someone’s changes I’m reviewing, continue my work while my own check ins are held up for some reason like a code review, or for keeping a compilable baseline around while the central one is broken.

If any of you centralized fans have ever created a branch at work for a common yet useful purpose like that, I would love to hear about it. For some people, even if the policies allowed it, the simple fact that everyone can see those branches would often prevent them from doing so. Nothing beats trying DVCS for yourself to experience the complete psychological freedom of being able to create as many branches as you want for whatever reason you want.

In conclusion, while the technical differences are not permanent, there are a number of social factors that will continue to give DVCS a large advantage in employing more powerful branching models.

Why DVCS is better with binaries than you might think

July 17, 2010 8 comments

Almost every discussion about distributed version control systems (DVCS) on the Internet includes at least one post along the lines of “DVCS can’t merge binary files and my project has binary files.” There is a good reason why you might not want to use DVCS for binaries, but contrary to popular belief, DVCS not being able to merge them isn’t it. My purpose here is to try to convince you why, with some exercises you can try yourself at home.

Most of the confusion arises from not understanding the difference between merging binary files and merging branches containing changed binary files. The former no version control software can do, and the latter DVCS can do just as well as any other VCS, if not better.

I can already feel the heat from people preparing their replies. This is one of those things that people will argue about forever even when they’re wrong, because it seems so obvious that they don’t need to bother trying it for themselves. In anticipation of this, I’ve prepared a short demonstration.

Here’s the scenario: Bob is working on updating the documentation on his company’s website for their upcoming software release. Bob branches from the staging branch so he can work on updating the screenshots to move the minimize/maximize/close buttons to the left. It takes Bob longer than the original estimate, because he didn’t realize you could just take new screenshots instead of editing the old ones in Gimp, but he eventually merges his branch back into the staging branch.

At this point we have one branch that says buttons go on the left and one that says buttons go on the right. Obviously a merge conflict, right? Wrong. The merge algorithm knows the screenshots haven’t changed in the staging branch since Bob branched off and does the right thing. Don’t believe me? Try it for yourself. I’ll wait.

Some people get it in their head that this will work for text files, but not binary files, because binary files aren’t “mergeable.” Note that in this scenario, the merge algorithm doesn’t care if the files are mergeable or not.

That’s not to say there aren’t scenarios where mergeability matters, but with binary files you hope you never get into that situation, because no version control can get you out of it. If Alice is changing the same screenshots at the same time as Bob, there’s no way to merge them automatically.

To help out people who don’t like scary solutions like communicating with your coworkers, most centralized version control software lets you place a lock on a file. Because none of the major distributed software has this lock feature yet, people claim it’s because it’s fundamentally impossible with DVCS.

While it’s true locking requires communication with a central lock authority, there’s no need for that to be in the same place as everything else, nor is there a need to be in constant contact with that central authority. If people spent as much time implementing the feature as they do whining about the lack of it, every DVCS implementation would have had locking years ago.

As I mentioned, there is one good reason why you might not want to adopt DVCS for your binary files. Binary diffs tend to be larger than text diffs, and with DVCS all the history gets stored on every developer’s computer. However, you shouldn’t assume that every change will increase your repository size by 100% of the binary file size. In my test for the Bob scenario, it only increased about 36% for Bazaar. You also shouldn’t assume that all that history is being copied every time you make a new branch. All the major software lets you share the diffs between local branches, and although the initial checkout may take a while, after that only the changes are communicated.

In conclusion, if you have been avoiding evaluating DVCS because of the binary myth, you might want to give it a second look and actually try it out on your own files. You may still find CVCS to be a better fit, but at least that decision will be based on evidence. On the other hand, I think you have a good chance of being pleasantly surprised.

Why DVCS provides better central control

July 16, 2010 8 comments

I previously discussed how distributed version control systems (DVCS) can help with keeping the tip always compilable. DVCS is also useful in making sure the tip always passes a test suite, or maintains any other standard of quality. It does this by giving more control where control is needed.

When you first hear about DVCS, that statement seems counter-intuitive. How can a decentralized system give more control to the central authorities? The key is that by giving up some control where it wasn’t needed, you gain more control over the important parts, sort of like guarding a prisoner in a 10×10 cell instead of a 10 acre field.

In our company we have product-specific code for a number of embedded products, and a large base of code shared between products. Because developers typically only have the hardware for the product they are working on, someone who makes changes to the shared code can only test it on that one product. As a result, although breaking your own product’s build is quite rare, shared code changes that break other products’ builds are much too common.

So how can we set up our branches to mitigate this problem? The answer lies in examining who we want to have control over each branch. At the same time, we think about what our ideal log would look like.

We want the “official” branch for a product to consist of a series of successfully tested builds. We want to be able to take any revision of that branch at any time with confidence. Obviously, the person best suited to controlling that branch is the lead tester for the product. The log at that level would look something like this:

Here we have the log for a fictional Product A.  Notice we only have one person committing here, the lead tester for product A.  This responsibility could be rotated among all the testers, and could be enforced by only giving the product A testers write permissions on the branch, or more loosely enforced just by social convention.  The important thing to notice is that the test group has more control over the product’s official branch than the typical centralized model, where all developers have commit access.

Okay, so where do the developers come in?  Developers like to have control, but it looks like you just took a whole bunch of control away from them.  For that we expand the log to the next level:

At this level you can clearly see which features made it into each promoted build.  Developers for product A have full control over this development branch and can set permissions on it as they see fit.  This includes preventing the test group from writing to this branch if desired, because all they need is to be able to do is pull.  In other words, each group has full control over exactly the areas they need it.  A developer’s view in their daily work looks like this:

This shows the changes for Product A as if Product A is the most important product in the world.  All the shared changes from Product B are hidden behind the plus sign, which you only click to expand if you want to see the details.  A developer on Product B would see a similar view of Project B’s development branch, as if Product B is the most important product in the world.

Here you can also see two possible approaches to receiving changes from the shared code.  One is what Amy did in revision 2.2.1.  For her change, she knew she needed some changes in shared code from Product B in order to proceed with her work, so she merged them in.  The other alternative is an assigned branch manager thinking it’s been a while since we synced up, so he more formally merges the changes in.  You can do both if you want.

Notice that either way the developers for product A have full control over when shared changes get pulled into their product’s build.  If the shared changes cause a product-specific compile or run time error, Amy simply doesn’t commit them until she has worked with the Product B developers to get it resolved.  In the mean time, Alice, Arnold, and the test team are all working from a clean baseline.

Another method we use to improve code quality is code reviews.  We use an online collaborative review tool, and it generally takes a few days to finish one.  In the mean time, you start work on your next change, going back as needed to fix the defects found in the review.  Turns out DVCS is useful in this situation as well, because we can create a new branch for our work as soon as the review is started, like so:

In conclusion, distributed version control offers many flexible ways to increase build stability through more local control and judicious design of a branching model.  What branching models have you found to be successful?

Why more incompatible software might be easier to learn

July 14, 2010 2 comments

I’ve noticed an interesting phenomenon during this development cycle at work.  I’m working on a product that has 97% functionality in common with one of our older, more established products.  The differences, few though they may be, are necessary and not insignificant.  What I’ve noticed is because there are so few differences, people assume they can understand the product without learning the differences, then get frustrated and confused when those differences don’t behave exactly like the older product.

The interesting part is I’ve worked with these same people on other projects with much more extensive changes, and in general they have been able to follow them easily.  In other words, 50% changes are no problem, but 3% changes throw them all off.

It reminds me somewhat of uncanny valley. I wonder if it means certain open source software would have more success by completely breaking from their commercial counterpart’s interface rather than be a 97% clone.

Where else have you observed that principle in action?

Categories: User Interfaces

Why the Tip Should Always Compile

July 12, 2010 7 comments

Having a source control tip that always compiles is one of those software truths that I thought was self-evident, but today was reminded that it’s not self-evident for everyone. I was ready to check an important change, and as per our process, updated to the latest to make sure everything still worked before checking in. As it turns out, the guys one floor up were not so courteous. The build was broken in several places.

It took a few minutes to determine the offender, and I shot him a quick email in case he wasn’t aware of the issue. “Sorry for the inconvenience,” came the reply. “We’re not done checking it in yet.”

Multiple people coordinating check ins can be complicated, so I gave them another hour. Still didn’t compile, but for slightly different reasons. I then had a four hour training class. I came back from that expecting to be able to check in before I went home, and they had made progress but the build still didn’t compile.

At this point it was clear this wasn’t just a matter of coordinating check ins. They were using the main branch the entire building shares to integrate and debug their changes with each other. In case it’s not clear by now why that is bad, we will probably not have a working daily build tomorrow. If we do have a daily build tomorrow, a lot of important changes will not have made it in. Everyone’s testing will be set back by a day because a small group thought they would save a little effort by circumventing the process.

Early in the development cycle, this might not be a big deal, but we are very close to release.  If there’s one thing I’ve learned about software, it’s that integration and debug time is hard to predict.  Don’t ever think “just this once” you’ll check in something broken because you “know” it will only take a few minutes to fix.  Dealing with accidental breakage is difficult enough.  Having an unusable repository for 6 hours cannot be classified as an “inconvenience.”

On the other hand, it’s difficult to blame them for being tempted, given the source control tool we’re using. We’re evaluating alternatives, but in the mean time are stuck with what we’ve got. Take a look at the following characteristics of our version control process. If they look familiar, you might want to consider a version control change of your own.

  • Merging is difficult, so we have one big branch that everyone checks in to.
  • We have rules like don’t check in things that don’t compile, but no technological way to ensure they are followed.
  • We can either push our code to everyone in the building or no one. There is no in between without a lot of manual work.
  • We can check in even if our local copy isn’t updated to the latest.
  • We have no easy way of cherry-picking only code that is known to work.
  • There is no easy way of collecting related changes, then committing them all to the main branch in one operation once they are integrated.

If that list sounds familiar, and you haven’t looked at distributed version control yet, now is a good time to do so. At this point, we are simply struggling to maintain what I consider a bare minimum standard of having a tip that always compiles error free. I haven’t even touched on the ideal of having a tip that always passes a test suite. If your tests are automatable, your tools should be able to be set up to automatically reject changes that cause it to fail, same as a bad compile. If, like us, circumstances necessitate tests being run manually by a human, distributed version control can help with that too, with the right branching model. More on that later.

Why globals should be avoided

July 9, 2010 Leave a comment

I thought avoiding globals was so widely accepted now that people just did it without thinking, then I came across a new one during a code review and marked it as a defect, assuming it was fairly self explanatory.  To my surprise, the author disagreed strongly, defending his code like it was his firstborn child.  The other option would mean putting it in a header file, and since so many other files included that header, he didn’t think that would be safe this close to release.

Well, yeah.  What part of “global” made you think a lot of code wouldn’t be affected?  And you thinking adding a global is safe, but a one line change to a header file is dangerous frankly scares me a little.  Okay, that’s what I wanted to say, but I was stymied because I’d accepted avoiding globals as a best practice for so long, I had forgotten why. Then I came across this reddit post and thought this was probably something we all could use a refresher on, so here goes.

  • Namespace pollution.  Inside our home, everyone knows “Michael” refers to my son.  Anywhere else, we have to be more specific.  Only use a global if you’re absolutely sure no one else in 500,000 lines of code will ever want to use the same name for a different purpose, like for system-wide exception handling or something.
  • Errors don’t get caught until linking.  If you’re using dynamic linking that could be a big problem, and even with static linking error messages aren’t as easy to track down as a compiler’s.
  • Bigger chance of a semantic mistake like thinking you’re using the same global in two places, when you’re actually using two with slightly different spellings.
  • Bigger chance of multi-threading issues with two threads accessing the same variable at the same time.
  • Harder to determine where that variable is defined and everywhere it is used.
  • Harder to change implementation details because a global could be used anywhere instead of just easily defined boundaries.

Anything I missed?  Ideas on circumstances where globals are acceptable?  Let me know in the comments.

Categories: C++, programming