A lot has been said in the last few years about distributed computing – that is, taking a problem and splitting it up into pieces that more than one machine (virtual or otherwise) can work on at once. It’s a powerful technique, but it doesn’t only work on software: your team can be distributed as well.
The co-located software team has it’s own unique problems, one of which is that many of the supposed advantages of co-location only work fully in what is called an “open plan” arrangement. Unfortunately, open plan arrangements are terrible for software development, as study after study have proven. Interestingly, the more experienced and effective your developers are, the less the open plan arrangement works, in fact.
There’s a fine art to building a team to do expert software development, and it’s not easy. Adding new team members is often a recipe for going slower, not faster. You not only need to get the new team member up to speed on the specific problem, but you have to try to make sure you’ve got the right kind of developer, one who won’t slow down a project even once they’re up to speed, or force the whole team to try to work at a lower level of technical expertise so they can keep up.
Of course, with traditionally co-located teams, you’re limited to the developers you can add. They have to be near enough to commute to your offices (where you must have space to have them). When you’re dealing with high-end software projects, this is a limited supply, even if you’re fortunate enough to be in a developer-rich location. Not all developers you might want to work with live in your city, of course.
What if you could build a development team of the best people for the project, no matter where they happened to live? That’s what we call a Distributed Team, and it’s a very different thing than a co-located development group, with different needs and different advantages.
A distributed team is one where some or all of the team members are not in close physical proximity to each other. They might be in a different room, in a different building, city, or nation.
This gives you a pool of, effectively, all developers in the world to choose from. The probability of locating the right team members can be considerably higher in this case, although finding them is still a challenge.
What’s in it for you?
As Jason Fried, in his book “Remote”, describes, there are huge potential advantages in this approach. He also touches on the fact that the organization can benefit hugely as well, and that’s what I’d like to concentrate on here for a moment. If your company decides to embrace remote teams, it has the opportunity to create a group that goes past the limits of geography, yes, but it’s more than that.
Because the management dynamic of a remote team has to change, it is necessary to judge team members input by results not by duration. In other words, if you can’t see butts in a chair for eight hours a day, you need to find a different way of determining if things are getting done – by measuring the results directly. Suddenly you discover that this is what you should have been measuring all along, as opposed to worry about how many hours a person is putting in, you can think about how much value your company is putting out instead.
Distributed teams have also nothing in common with the practice of “outsourcing”, even though that has been associated with an overseas team. WIth outsourcing, you’re asking another team, who are themselves co-located, just usually in a different country, to do your development task for you. If the development that needs to be done is unimportant or trivial, perhaps that’s acceptable, but if it’s core to your business, this always ends badly.
The idea that a project can be shipped overseas and done by the lowest bidder might sometimes work in manufacturing, but software development has almost nothing in common with manufacturing, so it’s basically a train-wreck, and most companies that have tried understand this by now.
Distributed teams aren’t the “cheap” way of doing a project, and they have nothing to do with lowest-common-denominator thinking.
Why isn’t everyone doing it?
Interestingly enough, distributed teams are in fact far more common than most people think, both in the commercial software space and outside it. The vast majority of open source software, for example, is built exclusively by distributed teams, many of whom have never even met each other. Nonetheless they have built some of the most successful software in the world, including the Linux operating system, the Apache web server, and tens of thousands of others.
Industry, however, has been slow to adopt the idea of distributed teams, for a number of reasons. For some it’s the fear of loss of control, as indeed the management technique for a distributed team is quite different, by necessity. For others it’s the bad taste in the mouth from the wave of negative “outsourcing” experiences, which as we described above really have nothing to do with a distributed team, but the idea of developers who are not physically at your company doing development work gets unfairly mixed in.
What’s it Like?
I’ve worked on and with distributed teams for many years now, and I’ve seen them done well and I’ve seen them done poorly.
A distributed team that works is made up of self-starters, developers that know their tools, know their trade, and understand how to deliver value steadily. Someone just learning to program is usually not a good candidate, although it’s quite possible to introduce beginners into an already successful team.
A critical mindset shift has to take place in any successful distributed team: the group design, communication, interaction and general energy that makes a team a cohesive whole and not just a dissociated group of individuals has to happen online.
Usually this takes the form of an online chat mechanism, either with or without audio and video as an adjunct. It can be something as straightforward as irc, which serves as the backbone for many a distributed team, all the way up to more sophisticated solutions like Sqwiggle or iMeet.
With the Apache Foundation, which has been run by distributed teams for many years, there’s a rule that “if it didn’t happen on a mailing list, it didn’t happen” – this simply means that any conversations that don’t get recorded online in a way that everyone can see them are not to be used for decision-making. For the foundation, the “place of record” is the project’s mailing list. For other remote teams, it’s been places like Campfire or FlowDock, or a forum like Discourse, but the critical thing is that there is one well-known “place” online that every team member, no matter where they are, can see what’s being talked about, even conversations and/or decisions that happen while they’re not online.
The tool chosen doesn’t matter so much as that mind-set, and once everyone on the team makes that transition, it starts to not matter much where people are.
This is often easier in a fully distributed team, e.g. where ever team member is remote from each other, as it becomes a necessity to use on-line communication in any case, so there’s no opportunity for the team members not “in the office” to get left out of the loop.
Once that happens, the remaining tools that a team picks up to share the work are less critical, and the choices that are available today are quite good.
A wide array of tools are available for remote development to make the process easier.
It’s important to start with the basics when selecting tools, and not to try to adopt too much at once. Starting with an online communication system such as IRC and Email at first, then adding other tools as needed.
The Apache foundation has done just fine for decades with little more than a mailing list server as it’s primary collaboration tool, so it doesn’t take a complex tool to get started.
It should go without saying, but any distributed project should select and use a good source control system. By “good” I exclude most proprietary systems – the least you should settle for is SVN, and git or Mercurial would be greatly preferred.
Tmux (or Screen)
Beyond the fundamental tools of communications and source control, true real-time pairing needs some kind of collaborative editor or screen-sharing. One of the oldest and most straightforward of such tools is GNU Screen.
Although it’s initial purpose was for managing sessions on a single host, another function of screen is to share sessions, which is where it’s potential for remote development comes along.
A more recent derivative of Screen called Tmux is actually used more often today than it’s predecessor, but the principles are identical.
Screen-sharing solutions that work in a browser window tend to be a bit slower than the thick-client alternatives, but more convenient. They are particularly useful when quick screen-shares are needed with a non-technical user on the other end. Product demos and such are good examples here, not to much for development pairing.
Editors especially designed for collaborative editing, such as SubEthaEdit, are another choice, and have some of their own advantages. Plugins such as Rudel (for Emacs) and Floobits (for vi, Emacs, Sublime Text and others) allow you to have your own editor, and to have it configured as you prefer, while still sharing some editor windows or tabs with your partner. Collaborative editors are a bit similar to Google Docs collaboration.
A hybrid between screen-sharing and collaborative editing is the service/App ScreenHero – it shares the screen of one person or the other, but both get a cursor, making collaboration easier than having to swap back and forth the “presenter”, like other screen-sharing solutions sometimes require.
Beyond sharing an editor tab or screen, the developers that are pairing need more fluid communications than text chat can provide. Although regular phone is certainly an option (and might be indicated if bandwidth is at a premium), a more popular choice, and one that is even built-in to some solutions, is VOIP (voice-over-I.P.). Typically this involves using a headset for adequate audio quality – a desktop mic is an option, but the built-in microphone and speakers on most laptops are definitely not adequate.
Considered superfluous by some developers, the value of video should not be underestimated. Humans communicate a lot with gesture, expression and body language, and adding this valuable channel to the communication between developers can help enormously. I recommend video whenever it is available, unless bandwidth makes it impossible. Some services, such as Sqwiggle, actually provide an “always-on” video experience for distributed teams, and this has some merit as well.
Much like audio, an external webcam is often superior to a built-in laptop webcam.
Over and above the requirements of remote pairing, all of which are nicely met with Tmux or screensharing (plus some form of audio), it’s often helpful to be able to share the view of a presentation for teaching purposes. Many collaboration solutions include the feature of presenting slides to other parties.
Services such as iMeet have presentation built-in, as do open-source alternatives such as BigBlueButton.
More valuable than presentations is the ability to share a whiteboard or diagram, as many developers are used to living in very close proximity to a whiteboard.
Even if the particular application you’re using for collaboration doesn’t have diagramming build in, apps such as A Web Whiteboard (http://awwapp.com/) can easily be accessed from any browser. Once you share an ID with your pair, you’re both able to see and edit an online diagram. A Web Whiteboard is more freestyle – if you need more formal diagrams, LucidCharts or Gliffy might be a better option.
One element that adds a more complexity than many others is the range of time-zones involved. One or two are no big problem, even three are quite manageable, provided people are willing to stretch their day a few hours one way or another.
Once you get six or more time-zones in the picture, though, you are best served by working fully asynchronously, e.g. to not try to pair in real-time with developers in two different zones. This also means your online text communications must handle asyncronous traffic – e.g. you might b e better served by something like Discourse, which is a bit more “forum-like” and supports threaded discussions than something like Campfire, which is more linear and well suited to realtime or close-to-realtime communications.
Another option is to use more technique and tools for code review, to adjust a bit for the lack of ability to pair in real-time. Tools such as Git pull requests, Gerritt, or Atlassian’s Crucible are candidates for this area, but they are only an aid, not a real substitute for real-time pairing with VOIP and video.
When you do have to span time-zones, a site like EveryTimeZone (http://everytimezone.com/) can be handy.
The one constant requirement of remote development of any kind is to have the right people. Remote development takes a slightly different grade of developer than in-person work does: you need people who have enough skill and experience to be able to work on their own (even when they are collaborating). They need to be responsible and self-organizing, as well as having a clear sense of the task at hand and the goals their development are supposed to achieve.
If you add time-zones to the mix, you need even better people – this is not a task for junior apprentice developers, and is a terrible place to try to save money: If you’re not paying your remote developers more then your in-house people (if you have both), you’re probably doing it wrong.
This article can’t do any more than give you a taste of the possibilities of distributed teams, and the tools and techniques for making them work, so we’ll go into some of the details in future posts.