Escaped Defects

I was tempted to inject a typo in this post just for the irony.  If you find it, post in the comments. ;-)

Software development is challenging.  In an agile world, baking quality in—defect prevention—is the norm, or at least it should be. Done well, this kind of development can be incredibly rewarding and the feedback loops are almost instantaneous.

When teams are test driving, the world of software development simply shows up differently. Working, Tested Product is the primary measure of progress and the value of new functionality for changes can be recognized by customers quickly. Automation and quality is central to making that possible, but not everyone has built out Continuous Integration (CI) and Continuous Delivery (CD) such that something built today can also be tested and released to Production on the same day.

Teams that have established Definitions of Done, (DoD) including code quality standards and testing standards and automation throughout enjoy a different existence than the vast majority of teams that I have encountered.

Agile nirvana is rare, at least in my experience.

For a lot of organizations, development is separate from testing and developers may not sit with or even collaborate with testers. Work moves like a virtual assembly line from one person to the next; one environment to the next. For many organizations, the journey from development to Production (left to right) looks similar to the diagram below. 

  

With multiple stages of testing through multiple environments--an assembly line approach to testing--an issue can arise in any one of them. In fact, when software is not test driven, finding issues in these environments can be considered the norm.

When things go wrong though, issues can cause organizations real discomfort. Escaped defects are a pain and it is critical to ferret them out before they get to Production. For some organizations an “Escaped Defect” is an issue that is found in Production, but I believe that sets the bar too low.  In my book, an escaped defect is any issue that is discovered after the development team is done with it. That means a separate defect is logged for anything found after it leaves the “Development Environments” depicted on the left in Green in the diagram above.

Escaped defects are a critical quality metric. The cost of addressing a defect increases significantly the later it is found. Remediation and then retesting are all incremental costs that basically double the cost of solution delivery. For every escaped defect, the organization is basically paying for the defect that was originally created and then paying again when it is remediated. Basically, at least twice the cost for any value that involves a defect.

In my view, those cost increase in a non-linear way.  The later a defect is found, the more backflow it produces for remediation. The cost of backflow means that not just the functional code has to be rebuilt, but the test code associated with it as well. Each phase of testing may have separate test code that is affected by the changed functionality. The cost can increase exponentially depending on the extent of the defect.

The solution of course is to never produce a defect. If only the real world were that simple. Learning to test drive and evolving a DoD can help a team continually raise the quality bar and move closer to defect prevention rather than a sole reliance on defect detection. As the team learns, the DoD can change to reflect the new capabilities. Any team that is just learning TDD should not start with a quality standard of 80% automated test code coverage. It is just not realistic, and automated test code coverage may not be the most critical metric to track. Whatever metric you choose, start where you are and then increase the standard as the team develops the capability to meet it.

There are three key take-aways I want you to get from this:

1.     Traditional defect detection is an expensive way to ensure quality. Defect prevention is the ultimate goal. Just think if we could only have had today’s forward-looking radar on the bow of the Titanic.

2.     Start as early as you can with Software Craftsmanship and adoption of Test Driven Development. Agile organizations are continually working toward bulletproof quality, full stack test automation, CI & CD, and the sooner you can get started the better.

3.     The goal is to not just “do Agile”, but to “be agile” and escaped defects can be used as a barometer for progress towards that agility. As the number of them goes down, agility goes up.

The road to agility can sometimes be long and winding. The quality of your code and your products is central to making it successfully. Declare war on your escaped defects! Track the amount of backflow you are experiencing by designating every issue that is uncovered after a team has declared a story done as an escaped defect. Live in the pain of the imperfection and use defects as a forcing function to raise the bar on quality.  Hemming and hawing about when a defect should be logged only muddies the waters. If a team said it was done and then more work gets uncovered later, declare it a defect and learn what needs to change in your process, skillset, tooling or whatever in order to prevent similar defects in the future.

Did you find the typo? I may not have had the heart to add it.

Happy Journeying.

tc