Bugs and Unit Tests

We’ve started running a “bugs postmortem” on some of the bugs we’re finding in our software. The aim is to stop similar situations arising again, and identify any weaknesses in our process which are allowing these bugs to slip through the net.

We talked about one earlier today when we were returning an incorrect statistic from certain types of analysis. I think this started failing when we changed a defined method in an obscure and distant header.

When discussing what could have been done to prevent this bug, I said that proper unit tests covering this number would have spotted the problem as soon as it arose. This is quite true.

However, I then started talking about the test infrastructure we’d need. We would need mock images, I said, and a library of customer data to draw upon. Then we could run the  whole analysis and spot the problem. Building up this infrastructure would take time, I said.

Shortly afterwards, it occurred to me that I was missing the point. The problem wasn’t in the whole software module; and our software is not one massive black box to be tested.

The problem lay within one very specific method in one file. We could (should!) have had unit tests to cover this method. It’s much easier to do; we can create a dummy object, then pass in a selection of inputs, and check the resultant output. No need for images, or anything more complicated, because we’re testing on the level of a single unit of code.

That’s the big difference we must get our head around – automatic testing of a whole module is difficult, but adding a unit test to cover a single, simple, clean method is pretty easy.

With a comprehensive set of unit tests, problems like this would rarely arise, and be quickly fixed, leading to better software, and less frustrated developers.

Posted in Uncategorized | Leave a comment

Epic Win

Our recent release is still ringing in our ears – and I must say it still seems like a good ‘un. Everything went smoothly, before, during and after. We’ve had a load of positive feedback, and seen a record low number of bugs and complaints. All in all, well done us.

Now that’s out of the way, we’re trying again with Story Points. To do this, we need to have a good idea of what a Story is, and why we should use it for planning and estimating future work. Here, I ramble on in an attempt to make sense of the concept, with the help of spreadsheets and Battlestar Galactica.

What is a User Story?

A Story describes an enhancement in functionality, from the perspective of the user.

So is a Story like a functional specification? Not quite. We’ve built up our own way of writing specifications over the past few years. A spec should be concise; it should describe a problem, and outline a solution; it should be clear and descriptive, as well as being a living document.

A Story sounds like it should be too vague to be of use, but I think the opposite should be true.

It should start from the perspective of the user. Fill in the blanks:

  •  – “As a user, I want to … so that I can …”

This keeps us focused on tasks that actually add value – if we can’t finish the question, we shouldn’t be doing the work.

A Story might describe a tool, perhaps, or a one-trick dialog. Let’s take an example:

  •  – As a user, I want to change the colour of numbers in my spreadsheet, so I can visually identify important columns and totals”.

This sentence doesn’t say anything about the implementation. Should we have a dialog? Toolbar? Brain-computer bluetooth interface, as seen in the Dr Who episode Bad Wolf? These are important design decisions, and should be considered and written down:

  • – There will be an item in the Tools menu, labelled “Change Colour of Selected Numbers”
  • – This will bring up a dialog which looks like this:

… and so on. Here we define behaviour at a carefully defined level. Not too much detail, but enough to convey the spirit in which the story is written.

Epics and Themes

Stories should be small, and so multiple, independent Stories come together to make an Epic. There’s no reason to assume an Epic should be completed in a single sprint. For example, we might want to consider the Implement Llama Font Story to be part of the Look Pretty Epic, which reads;

  •  – “As a user, I wish to manipulate the look of the numbers in my spreadsheet in order to clearly convey statistical anomalies to third parties.”

And similarly, multiple Epics come together to make an overriding Theme:

  •  – “As a user, I wish to visualise my data in innovative ways in order to clearly and effectively convince prosecutors of my innocence for the fraud of which I have been convicted.”

This hierarchy keeps our mind on the ball. If a decision needs to be made, and it’s not clear from the Story’s description what we should do, we can consider the dilemma in terms of the Epic framework, or even the overriding Theme, all of which are descriptions of how we can help the end user – the customer.

The terminology is a powerful metaphor; one with legs, I think. Let’s consider a Story, Theme and Epic from a rather different source:

  • Story: Starbuck shoots down a Cylon Raider, salvages its systems, and flies it back to Galactica.
  • Epic: The Battlestar Galactica protects a civilian fleet from multiple Cylon attacks.
  • Theme: Stranded humans seek Earth.

How does the Story impact upon the Theme? Well, it’s not clear at first; but it certainly advances the Epic, and when considering the details it becomes clear – without the Raider, Starbuck could never have retrieved the Arrow of Apollo. Obvious.

In the same way, changing the colour of a few numbers can never clear an accused spreadsheet-fiddler of fraud, but together with a whole bunch of other stories, it gets us in the right direction.

All Is Not Lost

We can implement a last-minute approach to mapping out Stories; it’s clear that Number Colour is more important than Llama Font, so the details of that story can wait for a week or two, Similarly, we decide the most important Epic, and fully define the most important Stories within it.

And here is the fundamental difference between Lost, and the far superior Battlestar Galactica remake – BSG was an Agile development, whilst Lost was not.

On the face of it, the production of both had Agile features. They both engaged in last-minute script writing, responded quickly to user feedback, and each episode provided more value (entertainment) to the customer in an incremental way.

But while Lost (arguably) had an overriding Theme or two, it lacked anything at the Epic level. The producers have recently (finally) agreed they were making up the Stories as they went along. Black smoke? Yeah, cool! Magic numbers? Great idea. But this is the televisual equivalent of developers adding the features they want to write. Add sound effects when your spreadsheet updates? Great, but where’s the value to the customer?

If we can get a grip on why we’re writing user stories, we can keep in our mind what we need to implement; when we need to do it; and how everything we do is advancing the user’s experience of our software.

And, just maybe, defend humanity from Cylon raiders, while we’re at it. (Next time: How Agile methodologies can help a Cylon Centurion buy a pint of milk from the corner shop.)

Posted in Uncategorized | Leave a comment

Floored Thinking

Like many others around the country, I spent the recent bank holiday weekend attempting a spot of DIY. And as I knelt on Monday afternoon, cradling the still-warm corpse of yet another terminally mis-jigsawed section of wooden flooring, I calmed my sobs and began to consider how I could turn around my DIY failures.

In my head, I took chalk to the still-unpainted wall and drew a scrum board. Post-it notes fluttered across; user stories popped into my head: “A visitor wishes to walk across the floor without spraining her ankle …”

Of course, I didn’t do this. I didn’t need a sophisticated engineering methodology – I just needed to be much better at DIY.

Step away from the power tools

Back at work, I’m a much better software engineer than I am a carpenter. That’s not saying a lot, but I have a dozen years of experience under my belt, and I’ve poured some of my best work into our products. And there’s plenty of room for improvement.

Developing our implementation of Scrum and Agile matters not a jot if our programming skills slide. One way the two might work well together is code reviews. We’ve talked about implementing a team-wide formal code review system, but never actually tried it. And unlike many of our recent innovations, I have been far from enthusiastic about attempting it.

A great idea

Why? After all, on paper, it’s a great idea. There’s a bunch of tools available to make it easy – we can put our code up for review by the whole team, who can add comments and spot any potential problems. The less experienced can learn from the more; while the more experienced can spot opportunities for improvement in our own code as well as each others’.

Our progress in other aspects can be reviewed, too. As we try to clean up and refactor our code, we can consider best practices amongst the team, and look for more opportunities to simplify and improve our code base. As time goes on, we’ll leave behind a trail of improved code, as well as developing important skills now and for the future.

As I say, on paper, it looks great. So why have I been so reluctant to try it?

Back to the floor

Let’s go back to my new flooring. I know I’m no great DIYer, but I put a lot of sweat, tears and blood into my hallway (the stains will come out with time). It’s mostly flat, runs almost parallel with the skirting boards, and hardly creaks at all. I’m proud of it, and when someone comes to visit, I want them to say, “Gosh, what an excellent new floor! You can hardly see the hammer dents!”. The last thing I want is a professional builder to come round and snigger at the edging.

And with the code I write, more so. I should be writing perfect code all the time – I certainly shouldn’t be introducing bugs, smelly hacks, or unnecessary complexity. My ego can just about handle the fact that my flooring is on public view, but it would prefer, at work, to be pouring its output, safe from prying eyes, into a thick, black, sealed box containing only my code.

My code?

And there’s the key point – my code. A key facet of agile development – indeed, any good software development environment – is that the code is not my code, it is our code. At a simplistic level, the company owns all my output, but more importantly, what I produce is something that we as a team have committed to.

Communal code ownership, and its sister, horizontal feature assignment, are things we have been striving to achieve in our Agile aspirations. It remains, however, difficult to assign ownership of my code – perfections as well as flaws – to the team. And yet, if I want to get the most out of code reviews, be they formal or informal, it’s one power tool I’ll just have to learn to master.


Posted in Uncategorized | Leave a comment

On sprints, marathons, and high-energy particle physics

I think, finally, I have understood the concept of Story Points, and the value they (might) bring to our sprint planning. I will attempt to prove this by explaining them in terms of our implementation of Agile Development, the London Marathon, and the hypothetical, undetected Higgs boson.

Recently we have abandoned Story Points, which we were using to estimate the amount of work we could commit to in a sprint. This involved the laborious task of estimating every feature, sub-case and bug in Story Points. Without a baseline understanding of the value of a Story Point, they hovered somewhere between “meaningless” and “hours pretending not to be hours”.

The flaw in my understanding was mostly short-termism. Story Points don’t tell you how much work you should commit to in _this_ sprint; but they do tell you the feature velocity of a team over longer timescales.

Another mistake was assigning Story Points to bugs as well as features. It’s essential to consider the complexity of bugs when committing to work in a sprint, so this seemed like a good idea; but the key fact to remember is that fixing bugs is not _advancing_ the software. You add no value, no story, to the customer’s experience by fixing bugs. You’re just putting back value that should always have been there.

The London Marathon

Consider a product’s feature velocity. Our product is a marathon runner, running through crowded streets on a marathon that never ends. As we add features to the product, we move forwards – Greenwich Park, Charlton, Woolwich passing by. The faster we can add features, the faster we run.

But buggy features are an illusion of progress. The crowd expects us to be approaching Cutty Sark, but in fact we’re barely over the Blackwell Tunnel. If we were to fix all those bugs in a single sprint, we would appear not to move at all.

Story Points, then, are an indication of how far each new feature will take us. If we implement 60 Story Points in a sprint, we’ll get as far as Tower Bridge; 30 will only get us as far as Greenland Docks.

The Higgs mechanism

The bugs in our software – the stuff we must continually deal with, or be disqualified from the race – is our technical debt. Our technical debt is like the Higgs field.

As everyone knows, the Higgs field – interactions within which are carried by the legendary, and so far purely hypothetical, Higgs boson – acts upon all fermions and essentially provide them with mass.

You understand the physics behind the Higgs boson, right? After all, 10,000 scientists collaborated to build the Large Hadron Collider; I bet fewer software engineers understand Story Points.

Just in case you don’t, an oft-quoted analogy is that of someone moving through a crowded cocktail party. An unknown guest – in this case representing a massless photon (or gluon) – can quickly step through the uncaring guests without impedance.

Margaret Thatcher, however, is famous and influential. When she steps into the room, the guests quickly turn to face her, talk, bow, scrape, ask for an autograph, or rail against the social injustices of the 1980s. It becomes hard for her to cross the room, not because of her fame (or mass) – but because of the interactions of the cocktail guests (Higgs field).

Back to London

Where were we? Ah, yes – approaching Canary Wharf. Our technical debt now relates to the fame of the runner. As she powers along, fans jog alongside, shake hands, and slow her down. These are the bugs we must deal with in the sprint; we’re reducing technical debt, but at the cost of overall velocity.

It’s not far until the Mall now. The marathon has taken 13 sprints to complete. But we also know that we have completed 260 Story Points worth of work. Our velocity is 20 Story Points per sprint – and we have also cleared much of our technical debt. As we cross the finish line, we can breathe a huge sigh of release.

But our runner gets no chance to rest – we must immediately fly to Switzerland to embark upon the Geneva marathon. We’ve mapped the course; there’s 300 Story Points of effort in this one. But it’s clear to us that this will take, at current velocity, 15 sprints. (And if we can improve our velocity, by shrugging off more technical debt, maybe less.)

So that’s the point of Story Points. Large projects, when accurately broken down and estimated in Story Points, can be related to development time with some degree of education. Our current velocity can be measured alongside, and affected by, our technical debt (and its integral – the speed at which we are gaining, or losing, mass / fame / debt).

And now I can see the point. So what do we do now?

  • estimate tasks in a sprint in hours, so we know what we’re committing to;
  • estimate future features in Story Points, so we can calculate our velocity;
  • reduce our technical debt, bearing in mind the effect it has on the above;
  • impact hadrons colliding at near light speed, and examine the resultant particle shower;
  • train for the London Marathon.

Hmm … one step at a time, perhaps. See you in Geneva.

Posted in Uncategorized | Leave a comment

Anyone remember 2001?


Being honest, writing software is hardly at all similar to firefighting, stuntsmanship, deep-sea diving, or steeplejackery. Aside from the lack of uniforms (and basic standards of fitness, natch) there’s relatively few times at which we’re called to question our own mortality.This week, alas, was one such time for me – it’s been exactly 10 years since I started working here.

10 years seems an awfully long time to stick with the same job in IT. That’s a decade – nearly a third of my life. In my first decade I went from a useless, crying, vomiting ball to a walking, talking human being with an aptitude for maths. (Ignore the cruel souls who say I’ve since reverted somewhat.)

What’s changed since 2001? The biggest change, from a work perspective, was definitely the takeover of my original employer four years ago. This marked quite a shift in many aspects – the two companies had quite different approaches to software, business, and working practices. It wasn’t easy to make this shift (and I could write more than one blog post on that subject alone), but a lot has changed both here and in the wider corporate environment. Most fundamentally, I still find myself working in a dynamic, risk-taking, fast-moving office, keen to innovate and push our software as far as it can go.

Sycophancy aside, lots more has changed – most importantly, surely, the fact that my desk is now roughly 150% larger than it was when I started. I also have 300% more monitors, and 500% more processor cores.

For the list-oriented amongst my readers, here are some stats. Since 2001 I have:

  • changed desk 4 times;
  • had 8-ish new computers;
  • worked on 10 different operating systems;
  • worked almost entirely on 1 product;
  • dabbled in 4 others;
  • seen 5 major releases;
  • seen dozens of minor ones;
  • been key in implementing Agile development in our team;
  • learned more about microbiology than I ever did in education;
  • published one paper;
  • changed operating system loyalty exactly once.

Personally, I have:

  • got married once;
  • had two kids;
  • learned to drive;
  • owned 4 cars, which I have driven roughly 80,000 miles;
  • bought a house;
  • spent about 3.5% of nights in a tent;
  • dressed up as a giant wasp-demon in Trafalgar square;
  • drunk about 4000 cups of coffee at work;
  • learned the Programmer Dvorak keyboard layout;
  • still not figured out how to use the phones at work.

Conversely, I have not:

  • changed my principal programming language (still good old C++);
  • used PowerPoint except under extreme duress;
  • started wearing a tie.

The world, of course, has seen:

  • three Prime Ministers;
  • two American Presidents;
  • too many wars;
  • the explosion of social media, wikipedia, twitter, and facebook;
  • nine series of Big Brother UK.

What keeps a programmer in the same job for so long? Whatever it is, it clearly exists in this office. Of the eight people in the room when I started, five of us have lasted a decade here.

For me, it’s the combination of a genuinely interesting subject matter, combined with the levels of respect, investment and enthusiasm which allows us to innovate. We don’t blandly follow Waterfall-model style specifications, but are encouraged to have an opinion and interfere at every stage of the product lifecycle. I think that’s vital both for a research-oriented product, and for a motivated team of programmers.

Plus there’s the not-wearing-a-tie thing.

What excitement does the next ten years hold? Will we have flying cars, personal jetpacks and space hotels? Anti-ageing pills, a complete understanding of the human genome, and a definitive mass for the Higgs boson? A cure for cancer? An end to war, hunger and suffering?

Well, I have a phone that can stream films from the internet, an eBook which holds the complete works of both Shakespeare and JK Rowling, and a twitter account that tells me when my favourite celebrities are having lunch … so I guess anything’s possible. See you then.

Posted in Uncategorized | Leave a comment

Guest Post: Samson the Rabbit; “My life in R&D”

This is a guest post by Samson the Rabbit.

I was honoured to be asked to contribute to Al’s blog, and I hope I can do it some justice.

My first couple of days in the office were spent sitting on the front desk, whilst people paid £1 to Guess The Name Of The Cuddly Toy. Eventually, with the contribution of the postman, the sheet was full, and Al was revealed as my new owner.

Daniela delivered me to Al’s desk, forgetting my name on the way – thanks. It’s Samson. Al is eyeing me warily; he’s reminded of my biblical namesake, and considers me an unwelcome harbinger of a haircut.

Al’s desk is dominated by four large monitors which cluster over two keyboards. A Mac and a PC, with a desktop background consisting of an Icelandic mountain range stretching seamlessly over all four monitors. He seems inordinately proud of this.

Aside from the computers, it’s not the tidiest desk in the room. There’s a cluster of unread cell biology and image processing papers in the corner, and a stuffed parrot lurking behind a monitor.

Most of the developers here are working on new features. Al tells me I’m not allowed to mention specific features until they’re complete – and maybe even then. “It’s important that we keep our options open,” he explains. “Until something’s complete, we don’t know how long it will take, or even if it’ll be successful.”

The point seems pertinent. This morning, he has spent most of his time with his headphones on, fingers fidgeting across the keyboard, and grumbling at the screen. The feature he’s adding is taking longer than expected.

“We’ve spent a lot of time recently reworking some of the software architecture behind it,” he tells me. “It’s supposed to make it easier for us to add new features in the future, and to maintain what we’ve got. But porting over the old behaviour is taking longer than we expected.”

Does that mean all the architecture work was in vain? Al laughs – no. He shows me two C++ classes which do the same task – one in the old version, one in the new. I’m no C++ expert, but I can see that the new one is much simpler. It has many more methods, and they’re all shorter.

“601 lines of code in the old, 217 in the new,” he tells me, irrelevantly, “and most of those 601 lines are duplication, or laborious listing of literal strings and version numbers. We’re not re-writing – we’re refactoring.”

Refactor is a word I hear a few times today. Rather than just copying code around, the team seems keen to leave it in better shape once they’ve touched it. Is it working? Al shrugs.

“It’s immediately easier to read,” he says. “Our method names make sense, and our variables are all – well, mostly – properly encapsulated. But the proof of the pudding is in the eating. Ask me again in five years’ time.”

At this point, Daniela stands up and mutters, “Scrum”. Six people stand, stretch, and walk over to the scrum board. It’s covered in post-it notes. Al speaks first; he moves a couple of post-it notes around, and mumbles some excuses for not having finished something. He prods at the post-it accusatorally.

“But I will have it finished by tomorrow,” he says.

The others move post-its around; someone asks for advice on fixing a problem. Only the programmers speak; something to do with pigs and chickens. Then the burn-down chart is updated, with a few more hours of work knocked off.

There’s a bit of paper above the board declaring, in big letters, “April A Sprint Goal: Functional Equivalence”. “That’s a new thing,” explains Al. “The idea is that by the end of this sprint we’ll be finished our re-architecturing, and be in a better position to add even more new features. I was kinda hoping to keep everyone focused on that goal.” Is it working? “Ask me in five years.”

After lunch, Al disappears off to some top-secret meeting about new features, leaving me to stare idly at the flesh-and-blood rabbits hopping around in the hedges outside. To be honest my mind is drifting; I’m told there’s two young kids who’ll give me a more entertaining life back at Al’s house. I hope they don’t chew my ears. I hate that.

Well, thanks for reading this far; thanks to Al’s colleagues for raising a few quid for charity; and thanks to Al for letting me write this. I’ll be back in five years to see how he’s doing.

Posted in Uncategorized | Leave a comment

The Bag of Holding

Or: how we stuff our Sprints

Any player of Dungeons & Dragons (or, for a different flavour of geek, nethack) would love a Bag of Holding. You can put in as much stuff as you like (coins, swords, unconscious friends) and yet it never gets full. No self-respecting adventurer should be without one.

We have a variation within our implementation of Agile Development – it’s the Bag of Stuff To Do. It’s similar to the Bag of Holding in that we can stuff in as much as we like, but like its evil twin the Bag of Tricks, there remains a non-zero chance of being badly bitten.

Stuffing the Bag

In our Sprint planning meetings we decide what to put in the forthcoming sprint. As I’ve blogged before, it’s difficult to plan this bit right. In theory we have a well-groomed backlog of work; we can siphon off the right amount and commit to getting it done.

I’ve also blogged on the potential for our daily Scrums to descend into blandness. This is particularly apparent when we’re working on different projects. One developer is working on the FRAP system; another on file export; two or three more on new measurements capability. There’s three stories here, all of which have been bundled together and put in the Bag of Stuff.

We’ve decided that these three things are our priority for this Sprint. Or rather, we’ve decided that a certain fraction of these are our priority. We’ve broken down larger tasks, packaged them up with User Stories, and are happy (ish) that we can get them done. All very clever, well done us.

Rummage Fail

Alas, our Bag of Stuff cannot hold anything large – and the larger the contents, the harder it is to squeeze them out. If we put in two ironing boards and a lampshade, chances are at least one of those will get stuck.

What we should really fill the bag with is sand. That way, we can …

No, wait, metaphor fail. Let’s talk actual tasks. What we should fill the sprint with is small tasks, that we can all pick up and complete. We’re quite good at this – we have a good parallel skill set in the team – but still, we’re dishing out big, sprint-long projects to individual developers and letting them get on with it.

What fits the “sand” metaphor? Well, we could do nothing but fix bugs. Bugs are (usually) small, equally hard for everyone and the whole team gets to work from the same pool of tasks. But we don’t have that many bugs.

So instead, we could focus the whole team on only one thing. Pick a task, and ensure everyone is working toward the same goal:

“After this sprint, our software will be able to make tea.”

Then everyone works on this task. No exceptions. If we make the effort to break “Make Tea” up into bite-sized, parallelisable chunks, we can keep our focus, and work as a team.

Even if a sprint is only two weeks long, we can concentrate on one feature and get it nailed.

Team Bag

There’s some problems, of course. Bottleneck tasks, which must be done in a certain order. Hardware – only one person can work on the FRAP system at a time. And bugs must be tackled, a tide of infestation that must be herded and culled each sprint. But bugs can be tackled whilst bottlenecks are squeezed through; and UI design doesn’t require the hardware.

In upcoming sprints, let’s not just stuff the work we’d be doing anyway into an awkwardly-sized bag. Let’s have a clear Primary Goal for the whole team, and write it on the Scrum Board in BIG LETTERS. Then we can tackle the goal as a team, and actually work together towards our product development’s top priority.

Posted in Uncategorized | Leave a comment