How to build amazing software

  by Allan Mertner | 19 Sep 2009 | Permalink | Software, Software Engineering, User Experience

If you have ever tried, you will know that it’s hard to write software that is truly amazing, when “amazing” means that the software works in a way that is truly helpful to its users rather than just featuring lots of neat features. Doing it requires that all of the necessary ducks are lined up properly…

We have just released Foundation 7.3, and I think that really is amazing software. I hope that some of our customers and users will talk and write about how it helps them, and in this post instead focus on the process that has allowed us to build and release it.

When viewed in summary, the secret to success is really quite simple:

• Find out what problems our potential users and customers have
• Find out how they solve those problems today and how important they are
• Articulate a synthesis of the problems and use scenarios
• Prioritise and group the problems based on both urgency and value
• Give your development team the freedom to solve the problems one at a time
• Engage in an ongoing conversation with your users about what you are doing

The first four steps are what Product Management does, and I wrote about the process of figuring this out almost exactly a year ago. I never did write Part 2 of that post, but will instead look at the engineering process we have evolved to take in the data from PM, and which is what allows us to continually make many of the hard choices that are necessary to complete real software.

The following slide is from our 7.3 kickoff-meeting back in February and outlines how the development process works:

All development work is done in 4-week chunks that we refer to as Sprints. During the first week, we have a small team of people think about the sprint problem and how it can best be attached – everyone else finishes up training, documentation, automate tests and fix outstanding bugs from the last sprint. The real meat of the sprint is in the 3 weeks that follow: this is where the whole team implements what is necessary to solve the problem at hand, based on the architectural and design work done during the prep week.

At the end of the sprint, we publish a pre-release on the web site along with updated draft documentation and blog or forum post that describes what is new and who would benefit from trying it out. Of course, those that actually use the pre-releases then get a chance to provide feedback that we can react on before the product is ready to be released as “Generally Available”. To achieve this quick turn-around, the pre-releases are not fully regression-tested and have no upgrade path, but are nevertheless from experience very solid. Internally, we run a system called bleedingedge, which constantly scans our internal infrastructure and is updated every hour. In comparison with that system, the pre-releases are made available at a very sedate pace :)

Instead of a sprint where we develop new features, we can instead take the product to Release. This means that we spend 4 weeks tidying up loose ends, fix outstanding bugs, review and finish the documentation and perform a complete regression test of the entire product – and at the end release a new version that includes our standard 1-step upgrade and everything else that is needed to quickly get our customers onto the latest and greatest.

Production Support is also known as 3rd line support and is where customer cases that requires intimate knowledge of the code are escalated to by Customer Support. We don’t have a permanent team doing this, but instead have a rota that allows everyone technical to take on this role every so often. When there are no cases to work on, they get to work on the sprint instead – and this routine has proven extremely healthy for the product quality as a whole, since it means that everybody who writes or tests code will be exposed to the kind of real-life customer cases that are raised if the quality is not high enough.

The meat of the development sprint, where the team spends 3 weeks developing a solution that is ready for pre-releasing deserves a little more explanation:

The most obvious thing is also what needs to be said all the time: The goal is to solve the problem, not to add features. Sometimes solving the problem means that we have to rework an existing feature, such as when we removed the Basket from Foundation and instead included that functionality in the new manual grouping feature in 7.3. To make sure the problem actually has been solved, it is critical to think about how it will be used and how it can be demo’ed, and to constantly question and iterate the solution to make sure it actually meets the needs of the user.

We also actively encourage demos to happen during development, to get quick feedback on a new feature. Often, developers and designers call impromptu demos at the projector in our common area, where they show off the latest work and get critique from the rest of the team. The demos are an enormous help in ensuring things get looked at before they are “finished” – where it’s expensive and often disruptive to change – and also means the team is aware of most of what is happening during the sprint, even the parts they don’t work on themselves, bringing a cohesiveness to the whole that is otherwise hard to achieve.

The actual work is scheduled as stories, but instead of scheduling all of the necessary stories up front, we start with an initial set (prepared during the prep week) and evolve this as our understanding of what is necessary and possible changes during the sprint. This lack of rigidity allows the development team to be really creative about solving the problem, and has on more than one occasion meant that we had to redo some of the initial work when it became clear that the original solution was much too cumbersome, or just not good enough.

A story is a unit of work that is meaningful to an end user, and we try to break the agreed solution into stories wherever possible, and then have one or two developers work on a story until it is complete, as shown on this slide:

Doing documentation on a Confluence-based wiki has also helped a lot. Developers and architects can add rough developer documentation in the right place and format, and can easily add comments to the final documentation as well where it appears in context and therefore is easier to process and incorporate.

Bringing all of these elements together has taken a while, so feel free to borrow those parts that can work where you work: This process is better than any other I have used in the past 20 years. It allows our crack development team to develop something that has real, tangible benefits to our users and do so in a way that empowers them to come up with the best possible solution in the time available. Sometimes we do extend the schedule a bit to get what we want, and sometimes we compromise on what we can get, but that’s the way with real life: a working process has to be flexible enough to accomodate it!

So there you have it: this in outline is how we prodcuce amazing software. There are of course many more details and things to consider than I have included here – if you are interested in more or have something to add, just drop me a line or leave a comment below.