Escaped Defects

I was tempted to inject a typo in this post just for the irony.  If you find it, post in the comments. ;-)

Software development is challenging.  In an agile world, baking quality in—defect prevention—is the norm, or at least it should be. Done well, this kind of development can be incredibly rewarding and the feedback loops are almost instantaneous.

When teams are test driving, the world of software development simply shows up differently. Working, Tested Product is the primary measure of progress and the value of new functionality for changes can be recognized by customers quickly. Automation and quality is central to making that possible, but not everyone has built out Continuous Integration (CI) and Continuous Delivery (CD) such that something built today can also be tested and released to Production on the same day.

Teams that have established Definitions of Done, (DoD) including code quality standards and testing standards and automation throughout enjoy a different existence than the vast majority of teams that I have encountered.

Agile nirvana is rare, at least in my experience.

For a lot of organizations, development is separate from testing and developers may not sit with or even collaborate with testers. Work moves like a virtual assembly line from one person to the next; one environment to the next. For many organizations, the journey from development to Production (left to right) looks similar to the diagram below. 

  

With multiple stages of testing through multiple environments--an assembly line approach to testing--an issue can arise in any one of them. In fact, when software is not test driven, finding issues in these environments can be considered the norm.

When things go wrong though, issues can cause organizations real discomfort. Escaped defects are a pain and it is critical to ferret them out before they get to Production. For some organizations an “Escaped Defect” is an issue that is found in Production, but I believe that sets the bar too low.  In my book, an escaped defect is any issue that is discovered after the development team is done with it. That means a separate defect is logged for anything found after it leaves the “Development Environments” depicted on the left in Green in the diagram above.

Escaped defects are a critical quality metric. The cost of addressing a defect increases significantly the later it is found. Remediation and then retesting are all incremental costs that basically double the cost of solution delivery. For every escaped defect, the organization is basically paying for the defect that was originally created and then paying again when it is remediated. Basically, at least twice the cost for any value that involves a defect.

In my view, those cost increase in a non-linear way.  The later a defect is found, the more backflow it produces for remediation. The cost of backflow means that not just the functional code has to be rebuilt, but the test code associated with it as well. Each phase of testing may have separate test code that is affected by the changed functionality. The cost can increase exponentially depending on the extent of the defect.

The solution of course is to never produce a defect. If only the real world were that simple. Learning to test drive and evolving a DoD can help a team continually raise the quality bar and move closer to defect prevention rather than a sole reliance on defect detection. As the team learns, the DoD can change to reflect the new capabilities. Any team that is just learning TDD should not start with a quality standard of 80% automated test code coverage. It is just not realistic, and automated test code coverage may not be the most critical metric to track. Whatever metric you choose, start where you are and then increase the standard as the team develops the capability to meet it.

There are three key take-aways I want you to get from this:

1.     Traditional defect detection is an expensive way to ensure quality. Defect prevention is the ultimate goal. Just think if we could only have had today’s forward-looking radar on the bow of the Titanic.

2.     Start as early as you can with Software Craftsmanship and adoption of Test Driven Development. Agile organizations are continually working toward bulletproof quality, full stack test automation, CI & CD, and the sooner you can get started the better.

3.     The goal is to not just “do Agile”, but to “be agile” and escaped defects can be used as a barometer for progress towards that agility. As the number of them goes down, agility goes up.

The road to agility can sometimes be long and winding. The quality of your code and your products is central to making it successfully. Declare war on your escaped defects! Track the amount of backflow you are experiencing by designating every issue that is uncovered after a team has declared a story done as an escaped defect. Live in the pain of the imperfection and use defects as a forcing function to raise the bar on quality.  Hemming and hawing about when a defect should be logged only muddies the waters. If a team said it was done and then more work gets uncovered later, declare it a defect and learn what needs to change in your process, skillset, tooling or whatever in order to prevent similar defects in the future.

Did you find the typo? I may not have had the heart to add it.

Happy Journeying.

tc

Helicopters helped me learn to think ahead

When you learn to fly helicopters in the military, the instructor pilots regularly roll the throttle off at the most inopportune times, forcing you to autorotate. Back in my day, many times, we would do touchdown autos. Most of my instructors were old grizzled VietNam veterans who seemed to take sadistic pleasure in catching us off guard and seeing how we react to the simulated emergency. Helicopters are more reliable these days, but my dad said when he started flying them in the early 1960’s, they were not. The old saying goes “helicopters don’t fly, they beat the air into submission” and their story was that they were preparing us to deal with the “stress of combat”. The ultimate effect was that we learned to always be looking for where we could land should the engine actually quit someday.

There is a lot to consider when you autorotate and you have to think quickly. Your altitude, the direction and speed of the wind, possible landing spots, the weight of the aircraft, are all important factors in whether you and your crew will walk away or not. The instructors liked to get us really busy inside the cockpit with radio calls or navigation or something else and then roll the throttle off to see what we would do.  We would have to quickly see if there were any openings in the trees and figure out the safest place to try to land. If, when you looked out to see where you could land, there were no obvious quick options, you had to start to guide the aircraft towards what looked like the best option and then look for wind indicators, airspeed, shape your descent to align with the longest part of the clearing, make your mayday calls and never ever stop flying until you were on…..the……ground.

Some folks have problems though. It is easy to get fixated not on the clearing where it could be safe to land, but rather on the obstacles. The obstacles draw our attention and if we are not diligent, they can cause us to crash. There is not a lot of time to get focused on distractions when you are coming out of the sky like a green Mosler safe. Some folks were not able to overcome it and they washed out. If they couldn’t see their way to making it happen, to zeroing in with all their focus on the safest spot to land and then willing the aircraft there, then they wouldn’t last long in the program. So, there were folks who could see where they needed to go and do whatever it took to get there and then there were folks who had stuff happen to them and they responded like a cork in the ocean being battered about by the waves.

Leaders tend to bend the spoon. I see this again and again working with entrepreneurs. They tend to make things happen in their lives and the lives of others rather than simply respond to whatever life throws at them. Leaders have a clear vision of who they are, what they care about and conversely, what is not worth worrying about. They have goals. They have ambition, and they overcome obstacles as a normal course of getting things done. They are moving with a purpose and they never ever stop.

As human beings, we have the gift of language and we live in the stories we tell ourselves. In every moment, we choose the clearing or the obstacles. We can shape the story in our heads and then bend the world to fit that story. Only humans have this ability and whether we are proposing to “land a man on the moon and return him safely to the earth” or getting a cup of coffee, we own that story and the outcome. Situations can change quickly, but nobody should let go and just see what happens. We can and should make the difference for ourselves and others. Maybe those old instructor pilots were wise as well as sadistic. It was a painful lesson, but they taught me to keep looking ahead and thinking through all my options.