Before deployment, people might need to provide multiple information. For example, which nodes to deploy what services, use which tcp ports to listen on application endpoints, etc.
Even very careful person would make stupid mistakes! e.g, wrong ip format, invalid port, unsupported OS version, machine doesn’t have RAM, etc.
These human errors may not only fail your deployments, but also cause unexpected damages to your existing envs. Even mess up critical envs sometimes. So it’s better we enforce pre-check before update.
Original Article: https://dennyzhang.com/enforce_precheck
Connect with Denny In LinkedIn Or MailList
One-button click deployment is nice. Lots of DevOps practices drive this trend.
I usually enforce it via Chef + Jenkins:
- Core deployment is implemented by Chef
- Wrap up it as a jenkins job.
Consequently people only need to trigger the jenkins job with all parameters default.
Here is an example of Jenkins job:
Super easy, isn’t it?
Just one issue. The parameters!
People need to configure server_list and chef_json for correctly. It’s a bit complicated.
Compared to documentation, it’s better we enforce pre-check. Very few people enjoy lengthy manuals. So try your best to help people to avoid common pitfalls and human errors.
What Pre-Check To Enforce?
It depends what things you need people to provide.
For your reference, my favorite checks usually fall into below three categories:
- Parameters Check:
| Check | Summary |
|---|---|
| check_ip_format | No invalid ip format |
| check_ip_list_format | IP list |
| check_tcp_port_format | Valid TCP port |
| check_int_format | Valid integers |
| check_string_not_empty | Parameter is set |
- Infra Check:
| Check | Comment |
|---|---|
| check_capcity_disk | Enough disk capacity |
| check_capacity_cpu | Enough cpu core |
| check_capacity_mem | Enough RAM |
| check_os_version | Supported OS version |
- Network Check
| Check | Comment |
|---|---|
| check_ip_reachable | Verify network assumption |
| check_port_connect | Telnet server port |
You can implement your own checks. Then integrate them into your automation scripts as the very first part.
Feedback appreciated!
Re-use Code
Apparently different projects may all need to enforce pre-check.
To lower code duplication, I recommend you to implement a common library.
Here is what I’m using in Chef. (Check out in Github). Then different chef cookbooks can easily re-use this logic like below.
# Import library
Chef::Recipe.send(:include, PreCheck::HelperInputFormat)
Chef::Recipe.send(:include, PreCheck::HelperInfraCheck)
Chef::Recipe.send(:include, PreCheck::HelperNetworkCheck)
################################################################################
# Check input parameters
check_nodename_list_format(node['common_basic']['couchbase_hosts'])
check_nodename_list_format(node['common_basic']['elasticsearch_hosts'])
check_nodename_list_format(node['common_basic']['haproxy_hosts'])
check_nodename_format(node['common_basic']['nagios_server'], \
'nagios_server parameter is invalid')
check_tcp_port_format(node['common_basic']['app_https_port'], \
'app_https_port is not a valid tcp port')
check_tcp_port_format(node['common_basic']['haproxy_port'], \
'haproxy_port is not a valid tcp port')
################################################################################
# Check at infra layer
check_hostname(node['hostname'])
check_os_version(['ubuntu-14.04'], "#{node['platform']}-#{node['platform_version']}")
check_ip_reachable(['www.google.com'])
################################################################################
# Check hardware resource
check_capacity_cpu(node['common_basic']['precheck']['all_in_one']['min_cpu_count'], \
node['cpu']['total'])
total_memory = node['memory']['total'][0..-3].to_f / (1024 * 1024)
check_capacity_cpu(node['common_basic']['precheck']['all_in_one']['min_memory_gb'], \
total_memory)
Evaluate your deployment practice now. And share your thoughts and feedback with me, my friend!
More Reading: