6 False Negatives In Daily Deployment Test

We download files from external websites, however they’re in maintenance mode. We pull source code from Github/Bitbucket for build and deployment, but it’s temporarily down or unreachable. When code build, nodejs need to fetch community packages, java code need to download jar modules from a public nexus server, however the external servers flip from time to time.

It’s always a good practice to replicate and serve files in servers under our own control, instead of 3rd websites. How to detect all outbound traffic in deployment. Another improvement allows no hidden dependency and to make dependencies crystal clear.

You may be familiar with actions in below.

# Install Package
package XXX do
  action :install
end

# Download raw file from Github
remote_file '/opt/devops/bin/backup_dir.sh' do
  source 'https://raw.githubusercontent.com/' \
         'XXX/backup_dir/master/backup_dir.sh'
  mode '0755'
  retries 3
  action :create_if_missing
end

Quite often changes of latest version bring incompatible issues. This gives us surprise for our deployment test or issues hard to detect and diagnose.

It’s better using a stable tag/branch/version, instead of head revision. For our own code, that’s easy to enforce this. However for the community/open source code, the story is different.

To better utilize test machines, we may keep running lots of simultaneous test jobs all the time. The machine may run into low free memory or CPU. And this will fail our test, with no blame on our code.

Even worse, OS may run into OOM issue (Out of Memory). This may crash our critical services like Jenkins/DB, which demands human intervine. Or it runs into kernel panic, which blocks us to ssh, and machine reboot is our final resort.

To avoid this, we can add precheck logic before running new test jobs. When OS runs into low hardware resource, stop launching any more test jobs.

Two parallel jobs may fail, when they both want to get exclusive access for the same resources. Typically there are two major conflicts of deployment test jobs. 1. Run the same job parallelly 2. Run different jobs with shared resource.

Common shared resource are:

• Global Environments, like JDK version or global variables
• Docker specific resources: container name, mounted volume, NAT TCP ports
• ssh private key file of different Jenkins Jobs
• etc.

Deployment test will be invalid and trouble shooting effort will be wasted, if the test env is not clean. For example, if “apt-get update” fail, “apt-get install” is doomed to fail. If people has deliberately removed some files or packages in advance, deployment may fail as well.

It’s better we perform test on envs with a fresh start.

Application may takes several minutes to start, while performing system initialization or waiting db cluster up and running. We need to wait and check our assumption, before testing. Otherwise we will get false alarm again. Tips: How To Avoid Blind Wait.

Blog URL: https://www.dennyzhang.com/false_negative