Enforce Pre-check Before Deployment

Posted by DennyZhang

Before deployment, people might need to provide multiple information. For example, which nodes to deploy what services, use which tcp ports to listen on application endpoints, etc.

Even very careful person would make stupid mistakes! e.g, wrong ip format, invalid port, unsupported OS version, machine doesn’t have RAM, etc.

These human errors may not only fail your deployments, but also cause unexpected damages to your existing envs. Even mess up critical envs sometimes. So it’s better we enforce pre-check before update.

Enforce Pre-check Before Deployment

One-button click deployment is nice. Lots of DevOps practices drive this trend.

I usually enforce it via Chef + Jenkins:

  1. Core deployment is implemented by Chef
  2. Wrap up it as a jenkins job.

Consequently people only need to trigger the jenkins job with all parameters default.

Here is an example of Jenkins job:
github_one_button_deployment.png

Super easy, isn’t it?

Just one issue. The parameters!

People need to configure serverlist and chefjson for correctly. It’s a bit complicated.

Compared to documentation, it’s better we enforce pre-check. Very few people enjoy lengthy manuals. So try your best to help people to avoid common pitfalls and human errors.

1.1 What Pre-Check To Enforce?

It depends what things you need people to provide.

For your reference, my favorite checks usually fall into below three categories:

  • Parameters Check:
Check Summary
checkipformat No invalid ip format
checkiplistformat IP list
checktcpportformat Valid TCP port
checkintformat Valid integers
checkstringnotempty Parameter is set
  • Infra Check:
Check Comment
checkcapcitydisk Enough disk capacity
checkcapacitycpu Enough cpu core
checkcapacitymem Enough RAM
checkosversion Supported OS version
  • Network Check
Check Comment
checkipreachable Verify network assumption
checkportconnect Telnet server port

You can implement your own checks. Then integrate them into your automation scripts as the very first part.

Feedback appreciated!

1.2 Re-use Code

Apparently different projects may all need to enforce pre-check.

To lower code duplication, I recommend you to implement a common library.

Here is what I’m using in Chef. (Check out in Github).

# Import library
Chef::Recipe.send(:include, PreCheck::HelperInputFormat)
Chef::Recipe.send(:include, PreCheck::HelperInfraCheck)
Chef::Recipe.send(:include, PreCheck::HelperNetworkCheck)

################################################################################
# Check input parameters
check_nodename_list_format(node['common_basic']['couchbase_hosts'])
check_nodename_list_format(node['common_basic']['elasticsearch_hosts'])
check_nodename_list_format(node['common_basic']['haproxy_hosts'])

check_nodename_format(node['common_basic']['nagios_server'], \
                      'nagios_server parameter is invalid')

check_tcp_port_format(node['common_basic']['app_https_port'], \
                      'app_https_port is not a valid tcp port')

check_tcp_port_format(node['common_basic']['haproxy_port'], \
                      'haproxy_port is not a valid tcp port')

################################################################################
# Check at infra layer
check_hostname(node['hostname'])

check_os_version(['ubuntu-14.04'], "#{node['platform']}-#{node['platform_version']}")

check_ip_reachable(['www.google.com'])

################################################################################
# Check hardware resource
check_capacity_cpu(node['common_basic']['precheck']['all_in_one']['min_cpu_count'], \
                   node['cpu']['total'])

total_memory = node['memory']['total'][0..-3].to_f / (1024 * 1024)
check_capacity_cpu(node['common_basic']['precheck']['all_in_one']['min_memory_gb'], \
                   total_memory)

Evaluate your deployment practice now. And share your thoughts and feedback with me, my friend!

More Reading:

linkedin
github
slack

PRs Welcome

Blog URL: https://www.dennyzhang.com/enforce_precheck

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.