A test team’s worst nightmare is having a flood of critical bugs discovered by customers at a new release of a product. It seems that you can never test too much nor can you test everything. So it is always a fine balance, driven by many factors, of when to stop testing and deliver the software to the customer. But invariably bugs escape—it is only the quantity and severity that we, as testers, can hopefully control.
First we can start with attempting to understand why bugs escape testing by looking at how we test in relation to how customers use our product. Concepts from complexity and chaos theories will be introduced in light of the variables used within testing. Then some ideas on how to do effective root cause analysis with the bugs that customers do find and report will be discussed. Additional notions on mitigating the escapes by reviewing the testing approaches and techniques will be illustrated.
The takeaways from this presentation include approaches to root cause analysis, inclusion of chaos into testing and thoughts on viewing our products through the customer’s eyes.