After a lot of effort and communication, finally the system deployment works! To guarantee a smooth deployment anytime, we enforce daily deployment test as a next step.
Surprisingly daily deployment doesn’t always succeed like we expect, even if no major changes*. More interesting, many failed tests are kinds of false negatives. So what are the obstacles? And how we can avoid them?
What Does False Negative Mean?* Ideally each test failure should be an improvement opportunity. But if you tend to do a repetitive blind retry for a certain failure, we can say it’s a false negative. Why? It indicates either you don’t care about this failure or you have no other way to improve it. Furthermore false negatives come out with two bad consequences: it takes time to check and retry; it breaks or populates normal tests.
The success of basic deployment logic is only the beginning.* Besides constant changes initiated from Dev team,there are multiple things you need to pay close attention, if you’re ambitious to deliver a reliable and smooth deployment.
Here are several typical False Negatives, from my first-hand experience.