List The Slowest Steps In Deployment

System deployment and upgrade usually need to perform many actions. If we can detect and improve time-consuming steps, we shall get a better customer satisfaction or a shorter maintenance time window. However tons of steps are involved, how we can easily examine and figure out the bottleneck?

Performance Troubleshooting


You may think, if we can make sure all critical actions log messages with same timestamp format, we could get the time elapsed for each step.

The answer is Yes and No. Usually deployment may run automation scripts of several components or modules. e.g some are in bash, some in Chef/Puppet/Ansbile, or even some in Python, etc. It’s hard to enforce the practice, especially for the timestamp format convention. The good thing is that every professional tool/engineer will do effective logging for all critical actions, if not all. So the missing part here is how to attach the unified timestamp.

Fortunately Jenkins has a useful plugin called Timestamper. It can add timestamps to the Console Output of Jenkins jobs.

Jenkins Timestamper Plugin

Here is the idea:

  1. Automate deployment procedure as a bash script. And run it as a Jenkins job.
  2. Enable Jenkins Timestamper plugin properly for this Jenkins job.
  3. Caculate time performance of each steps by parsing Jenkins Console Output line by line.
  4. Sort steps by time performance with a descending order.

For better user experience, I’ve defined a Jenkins job: DiagnosticJenkinsJobSlow. Follow @dennyzhang

Below is a real example for how it works.

Notice: You can find a live demo here.

github_DiagnosticJenkinsJobSlow1.jpg

github_DiagnosticJenkinsJobSlow2.jpg


Original URL: https://dennyzhang.com/list_slowest_steps

Connect with Denny In LinkedIn Or MailList

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.