Enforce Pre-check Before Deployment

Before deployment, people might need to provide multiple information. For example, which nodes to deploy what services, use which tcp ports to listen on application endpoints, etc.

Even very careful person would make stupid mistakes! e.g, wrong ip format, invalid port, unsupported OS version, machine doesn’t have RAM, etc.

These human errors may not only fail your deployments, but also cause unexpected damages to your existing envs. Even mess up critical envs sometimes. So it’s better we enforce pre-check before update.

Enforce Pre-check Before Deployment


Original Article: http://dennyzhang.com/enforce_precheck

One-button click deployment is nice. Lots of DevOps practices drive this trend.

I usually enforce it via Chef + Jenkins:

  1. Core deployment is implemented by Chef
  2. Wrap up it as a jenkins job.

Consequently people only need to trigger the jenkins job with all parameters default.

Here is an example of Jenkins job:

Enforce Pre-check Before Deployment

Super easy, isn’t it?

Just one issue. The parameters!

People need to configure server_list and chef_json for correctly. It’s a bit complicated.

Compared to documentation, it’s better we enforce pre-check. Very few people enjoy lengthy manuals. So try your best to help people to avoid common pitfalls and human errors.

What Pre-Check To Enforce?

It depends what things you need people to provide.

For your reference, my favorite checks usually fall into below three categories:

  • Parameters Check:
Check Summary
check_ip_format No invalid ip format
check_ip_list_format IP list
check_tcp_port_format Valid TCP port
check_int_format Valid integers
check_string_not_empty Parameter is set
  • Infra Check:
Check Comment
check_capcity_disk Enough disk capacity
check_capacity_cpu Enough cpu core
check_capacity_mem Enough RAM
check_os_version Supported OS version
  • Network Check
Check Comment
check_ip_reachable Verify network assumption
check_port_connect Telnet server port

You can implement your own checks. Then integrate them into your automation scripts as the very first part.

Feedback appreciated!

Re-use Code

Apparently different projects may all need to enforce pre-check.

To lower code duplication, I recommend you to implement a common library.

Here is what I’m using in Chef. (Check out in Github). Then different chef cookbooks can easily re-use this logic like below.

# Import library
Chef::Recipe.send(:include, PreCheck::HelperInputFormat)
Chef::Recipe.send(:include, PreCheck::HelperInfraCheck)
Chef::Recipe.send(:include, PreCheck::HelperNetworkCheck)

################################################################################
# Check input parameters
check_nodename_list_format(node['common_basic']['couchbase_hosts'])
check_nodename_list_format(node['common_basic']['elasticsearch_hosts'])
check_nodename_list_format(node['common_basic']['haproxy_hosts'])

check_nodename_format(node['common_basic']['nagios_server'], \
                      'nagios_server parameter is invalid')

check_tcp_port_format(node['common_basic']['app_https_port'], \
                      'app_https_port is not a valid tcp port')

check_tcp_port_format(node['common_basic']['haproxy_port'], \
                      'haproxy_port is not a valid tcp port')

################################################################################
# Check at infra layer
check_hostname(node['hostname'])

check_os_version(['ubuntu-14.04'], "#{node['platform']}-#{node['platform_version']}")

check_ip_reachable(['www.google.com'])

################################################################################
# Check hardware resource
check_capacity_cpu(node['common_basic']['precheck']['all_in_one']['min_cpu_count'], \
                   node['cpu']['total'])

total_memory = node['memory']['total'][0..-3].to_f / (1024 * 1024)
check_capacity_cpu(node['common_basic']['precheck']['all_in_one']['min_memory_gb'], \
                   total_memory)

Evaluate your deployment practice now. And share your thoughts and feedback with me, my friend!


Check our popular posts? Discuss with us on LinkedIn, Twitter Or NewsLetter.