Enforce Pre-check Before Deployment

Before deployment, people might need to provide multiple information. For example, which nodes to deploy what services, use which tcp ports to listen on application endpoints, etc.

Even very careful person would make stupid mistakes! e.g, wrong ip format, invalid port, unsupported OS version, machine doesn’t have RAM, etc.

These human errors may not only fail your deployments, but also cause unexpected damages to your existing envs. Even mess up critical envs sometimes. So it’s better we enforce pre-check before update.

Enforce Pre-check Before Deployment


Original Article: https://dennyzhang.com/enforce_precheck

Connect with Denny In LinkedIn Or MailList


One-button click deployment is nice. Lots of DevOps practices drive this trend.

I usually enforce it via Chef + Jenkins:

  1. Core deployment is implemented by Chef
  2. Wrap up it as a jenkins job.

Consequently people only need to trigger the jenkins job with all parameters default.

Here is an example of Jenkins job:

Enforce Pre-check Before Deployment

Super easy, isn’t it?

Just one issue. The parameters!

People need to configure server_list and chef_json for correctly. It’s a bit complicated.

Compared to documentation, it’s better we enforce pre-check. Very few people enjoy lengthy manuals. So try your best to help people to avoid common pitfalls and human errors.

What Pre-Check To Enforce?

It depends what things you need people to provide.

For your reference, my favorite checks usually fall into below three categories:

  • Parameters Check:
CheckSummary
check_ip_formatNo invalid ip format
check_ip_list_formatIP list
check_tcp_port_formatValid TCP port
check_int_formatValid integers
check_string_not_emptyParameter is set
  • Infra Check:
CheckComment
check_capcity_diskEnough disk capacity
check_capacity_cpuEnough cpu core
check_capacity_memEnough RAM
check_os_versionSupported OS version
  • Network Check
CheckComment
check_ip_reachableVerify network assumption
check_port_connectTelnet server port

You can implement your own checks. Then integrate them into your automation scripts as the very first part.

Feedback appreciated!

Re-use Code

Apparently different projects may all need to enforce pre-check.

To lower code duplication, I recommend you to implement a common library.

Here is what I’m using in Chef. (Check out in Github). Then different chef cookbooks can easily re-use this logic like below.

# Import library
Chef::Recipe.send(:include, PreCheck::HelperInputFormat)
Chef::Recipe.send(:include, PreCheck::HelperInfraCheck)
Chef::Recipe.send(:include, PreCheck::HelperNetworkCheck)

################################################################################
# Check input parameters
check_nodename_list_format(node['common_basic']['couchbase_hosts'])
check_nodename_list_format(node['common_basic']['elasticsearch_hosts'])
check_nodename_list_format(node['common_basic']['haproxy_hosts'])

check_nodename_format(node['common_basic']['nagios_server'], \
                      'nagios_server parameter is invalid')

check_tcp_port_format(node['common_basic']['app_https_port'], \
                      'app_https_port is not a valid tcp port')

check_tcp_port_format(node['common_basic']['haproxy_port'], \
                      'haproxy_port is not a valid tcp port')

################################################################################
# Check at infra layer
check_hostname(node['hostname'])

check_os_version(['ubuntu-14.04'], "#{node['platform']}-#{node['platform_version']}")

check_ip_reachable(['www.google.com'])

################################################################################
# Check hardware resource
check_capacity_cpu(node['common_basic']['precheck']['all_in_one']['min_cpu_count'], \
                   node['cpu']['total'])

total_memory = node['memory']['total'][0..-3].to_f / (1024 * 1024)
check_capacity_cpu(node['common_basic']['precheck']['all_in_one']['min_memory_gb'], \
                   total_memory)

Evaluate your deployment practice now. And share your thoughts and feedback with me, my friend!

More Reading:


Leave a Reply

Your email address will not be published. Required fields are marked *