Shaun Abram
Technology and Leadership Blog
Running Jenkins jobs sequentially
I am a huge fan of the Jenkins continuous integration tool, using it not just for continuous integration, but also continuous delivery, server monitoring and performance testing.
Although using Jenkins is normally a breeze, I recently had a need to run several Jenkins jobs sequentially and it proved a little trickier than expected, so I thought worth posting about.
Basically, I wanted to call a database backup job before calling the deploy build job, which obviously shouldn’t be done in parallel. After a bit of research, I found a number of ways to have Jenkins run jobs sequentially.
1) Chain jobs together
Chain Jenkins job together using the “Build other projects” option in the “Post Build Actions” of a job. The downside to this is that it introduces a dependency between jobs. For example, I want to be able to run the database backup job without doing a deployment every time.
2) Parameterized Trigger Plugin
When you have the Parameterized Trigger Plugin installed:
- Create a wrapper job for your sequential jobs
- For each sequential job
- Select Build->Add build step->Trigger/call builds on other projects
- Enter the sequential job name
- Check the ‘Block until the triggered projects finish their builds’ checkbox (this only appears when you have the Parameterized Trigger Plugin installed)
Now when you run the wrapper job, all the triggered jobs will be run in order and sequentially. You of course also have the option of using the parameters functionality of the plugin too.
3) Throttle Concurrent Builds Plugin
The docs for the Throttle Concurrent Builds Plugin seem to be a little lacking, but it works well. With it installed, a “Throttle Concurrent Builds” section shows up in the Jenkins config section (Jenkins -> Manage Jenkins -> Configure System). There, you create a “Multi-Project Throttle Category”
Next, for each job that needs to be run sequentially, you
- Check the ‘Throttle Concurrent Builds’ check box
- Select the ‘Throttle this project as part of one or more categories’ radio button
- Check the checkbox of the “Multi-Project Throttle Category” you created above
- Save
Finally, you create a wrapper job that triggers all your to-be-run-sequentially jobs and when you run it, you should see your jobs running sequentially and in order.
Although a fairly complex option, this approach also allows you to run some jobs sequentially and others in parallel. For example, your wrapper job can run jobs A and B sequentially and jobs C and D sequentially, but trigger A and C in parallel. You can also combine this plugin with the Parameterized Trigger Plugin for maximum flexibility.
4) Join Plugin
Although I haven’t used the Join Plugin myself, it seems worth mentioning. Its docs state that
This plugin allows a job to be run after all the immediate downstream jobs have completed. In this way, the execution can branch out and perform many steps in parallel, and then run a final aggregation step just once after all the parallel work is finished.
That’s it. Several options for running Jenkins jobs in parallel. Choose the one that best suits your needs…
Tags: agile, ci, continuousintegration, devops, hudson, jenkins
Thanks for the pointers!
Appreciate the more in depth research.
Here is another similar post
http://zeroturnaround.com/rebellabs/how-to-use-jenkins-for-job-chaining-and-visualizations/
It adds to the list of ways to chain jobs via:
* the build pipeline plugin
* Xtrigger plugin (from comment)
I would add new plugin as a solution – not strictly chaining but running multiple jobs and then a single job afterwards (so an alternative to the join plugin)
the multijob plugin
That’s just what I was looking for, thanks!
Thank you very much 🙂 This works perfectly!
I am create a project doing sanity test, the first test is to check all slaves are online, basically execute an “exit 0” command. Use Parameterized Trigger Plugin, I created a master job and 10 more jobs to test 10 slaves. To execute this master job, it requires 10 executors (I run this on master). This 10 executors looks a bit scary to other people. Is there a way to run this 10 jobs in sequence?