Shaun Abram
Technology and Leadership Blog
Subscribe to RSS FeedHow much is your slow lead time costing you?
In a previous blog post, I discussed slow build times and estimated the associated costs. The build process is only one part of getting software out the door however.
Lead time is the time it takes to go from code committed to successfully running in production. This will include the build time we covered in the previous blog post, as well as all the other things required to get your code into users hands such as testing & deployments. This article focuses on the costs of that lead time.
Using the example of a team of 10 engineers, I estimate that the costs of a slow (one week) lead time could be the approximate equivalent of more than 3 engineers, or $400,000 per year. And I think it’s entirely possible that is on the low side since there are other costs that are just difficult to estimate. Imagine how much more you could achieve with 3+ extra engineers on the team.
Charity Majors goes further (discussed below) and suggests that reducing the lead time to hours could save the cost of 5 engineers on such a team. I was initially skeptical on that claim, but after trying out these estimates, she think may well be more accurate that my possibly over-conservative math.
Thanks
A big thank you to my former colleagues Dave Taubler, Abhijit Karpe, Josh Outwater and Steve Mauro for providing feedback and input on this article.
Most of the feedback took issue with some aspect of the estimates, which is fair, but the common theme seemed to be that everyone agreed that there is a very real cost to slow lead times, that it is high, and that using data where you can and estimates where needed is a good way to surface and highlight that cost.
Tags: accelerate, ci, cicd, continuousdelivery, continuousdeployment, continuousintegration, deliverypipeline, devops, metrics, stateofdevops
How much is your slow build costing you?
Slow builds are a pain, but how much do they really cost? How do you compare the benefits of reducing your build times against a new user-facing feature that generates real revenue, for example?
Your slow build could be costing you up to $1 per minute per build per engineer, based on the estimates shown below. So, even before you factor in CI infrastructure costs, slow build times can very quickly add up. In the example below, a team size of 10, each doing 5 builds a day, and each with a 30 minute build time, we calculate the cost could be up to $375,000 per year in waste.
This post and the calculations used in it are based on the approach taken by “Prioritizing with Cost of Delay” by Jeff Palmer. “Quantifying the Costs of Builds“, by Hans Dockter @ Gradle, also covers some of the same ground in similar and more comprehensive ways.
Tags: build, ci, cicd, continuousdelivery, continuousdeployment, continuousintegration, devops, metrics
Book Summary: Accelerate
Accelerate: Building and Scaling High Performing Technology Organizations is a book by by Nicole Forsgren, Jez Humble and Gene Kim. It is a follow on from the State of DevOps Reports that Forsgren and Humble used to publish (and which I wrote about before in Development and delivery practices for team success). I highly recommend buying the book, but here are some chapter summaries for the highlights.
Tags: accelerate, books, cicd, devops, metrics, mttr, stateofdevops, summary
Development and delivery practices for team success
Most metrics for measuring developer productivity, such as lines of code or issues closed, are notoriously ineffective. But the research in the excellent State of Devops report shows that, rather than focusing on local metrics and individual developer performance, it is better to look at overall development and delivery practices. Specifically, there are metrics that predict and reflect a team’s ability to successfully deliver working software into production, including deployment frequency, and the mean time to restore service after an incident. This articles discusses why some metrics are useless, and takes a closer look at the recommendations in the 2019 State of Devops report.
Tags: accelerate, developerproductivity, devops, fourkeymetrics, stateofdevops
Report Summary: Accelerate State of DevOps 2019
This is an abridged version of The Accelerate State of DevOps Report 2019; essentially a cut and paste of the most salient parts. The original is about 18,000 words; This is about 2,500 words.
I highly recommend reading the original in its entirety, if you have time, and I’m a big fan of the Accelerate book too. As with all the other summaries I create, this just as as way to help me digest and understand an excellent article.
Tags: accelerate, devops, fourkeymetrics, jezhumble, nicoleforsgren, stateofdevops, summary
SRE vs DevOps
I’m really enjoying the Seeking SRE book. Chapter 12 covers SRE vs DevOps; a community sourced compare and contrast type discussion.
My favorite description is from Thomas Limoncelli, who suggested that:
DevOps engineers focus on the SDLC pipeline with occasional responsibilities for production operations. SREs focus on production operations with occasional responsibilities for the SDLC pipeline.
Tags: books, devops, seekingsrebook, sitereliabilityengineering, sre
“Ship It!” talk at the Boise Code Camp
Slides for my “Ship It!” talk at the Boise Code Camp in Boise State University today: https://www.slideshare.net/shaunabram/ship-it-boise
Tags: boidecodecamp, codecamp, devops, mytalks, production, shipit, talks
Continuous integration / Continuous Delivery / Continuous Deployment
The terms Continuous Integration, Continuous Delivery and Continuous Deployment can still confuse. Even Jez Humble, author of the book Continuous Delivery commented that there is “confusion around the terms continuous delivery versus continuous deployment and my own thinking and definitions have changed”.(1)
This is my attempt to distinguish, based on the sources listed below.
Tags: ci, cicd, continuousdelivery, continuousdeployment, continuousintegration, devops
Dynamically set Jenkins node
This post explains how to dynamically/programmatically set the node a Jenkins/Hudson job will run on. This is relatively easy to do statically, using the NodeLabel Parameter Plugin, but trickier to do programmatically.
Tags: ci, continuousdeployment, continuousintegration, devops, hudson, jenkins
Maven archetypes to create your project folder structure
Maven archetypes are useful for many things, including creating a folder structure to start with, even if you aren’t planning to use maven as your build tool. See a list of available archetypes here.
(more…)
Tags: archetype, devops, gradle, Maven, mvn
Running Jenkins jobs sequentially
I am a huge fan of the Jenkins continuous integration tool, using it not just for continuous integration, but also continuous delivery, server monitoring and performance testing.
Although using Jenkins is normally a breeze, I recently had a need to run several Jenkins jobs sequentially and it proved a little trickier than expected, so I thought worth posting about.
Basically, I wanted to call a database backup job before calling the deploy build job, which obviously shouldn’t be done in parallel. After a bit of research, I found a number of ways to have Jenkins run jobs sequentially.
Read more
Tags: agile, ci, continuousintegration, devops, hudson, jenkins