We’ve started running a “bugs postmortem” on some of the bugs we’re finding in our software. The aim is to stop similar situations arising again, and identify any weaknesses in our process which are allowing these bugs to slip through the net.
We talked about one earlier today when we were returning an incorrect statistic from certain types of analysis. I think this started failing when we changed a defined method in an obscure and distant header.
When discussing what could have been done to prevent this bug, I said that proper unit tests covering this number would have spotted the problem as soon as it arose. This is quite true.
However, I then started talking about the test infrastructure we’d need. We would need mock images, I said, and a library of customer data to draw upon. Then we could run the whole analysis and spot the problem. Building up this infrastructure would take time, I said.
Shortly afterwards, it occurred to me that I was missing the point. The problem wasn’t in the whole software module; and our software is not one massive black box to be tested.
The problem lay within one very specific method in one file. We could (should!) have had unit tests to cover this method. It’s much easier to do; we can create a dummy object, then pass in a selection of inputs, and check the resultant output. No need for images, or anything more complicated, because we’re testing on the level of a single unit of code.
That’s the big difference we must get our head around – automatic testing of a whole module is difficult, but adding a unit test to cover a single, simple, clean method is pretty easy.
With a comprehensive set of unit tests, problems like this would rarely arise, and be quickly fixed, leading to better software, and less frustrated developers.