How much is your slow build costing you?

Slow builds are a pain, but how much do they really cost? How do you compare the benefits of reducing your build times against a new user-facing feature that generates real revenue, for example?

Your slow build could be costing you up to $1 per minute per build per engineer, based on the estimates shown below. So, even before you factor in CI infrastructure costs, slow build times can very quickly add up. In the example below, a team size of 10, each doing 5 builds a day, and each with a 30 minute build time, we calculate the cost could be up to $375,000 per year in waste.

This post and the calculations used in it are based on the approach taken by “Prioritizing with Cost of Delay” by Jeff Palmer. “Quantifying the Costs of Builds“, by Hans Dockter @ Gradle, also covers some of the same ground in similar and more comprehensive ways.

What is a build

First, let’s start with the basics and clarify the terminology. Wikipedia defines a build as

“the process of converting source code files into standalone software artifact(s) that can be run on a computer”. (wikipedia.org)

The build process can include compilation, static code analysis & scans, version control (e.g. tagging in git), interactions with code repositories (e.g. pulling down dependencies & uploading newly built artifacts), and, crucially, running any automated tests too.

 

Cost components

 

When trying to estimate how much builds cost, there are 3 components that you could consider:

• Fixed build costs
• Variable build costs
• People (engineer) costs

Fixed build costs maybe something like a fixed price gitlab account where you are within the allocated CI/CD minutes per month allocation. Or perhaps the fixed costs associated with an internally hosted Jenkins install.

Variable build costs might apply instead of or in addition to a base fixed cost. For example if you are using CircleCI credits or JetBrains Space CI Credits.

 

Calculating the cost of waiting

 

For this article, I am going to focus on the harder to calculate (and often most expensive) part, the cost of the time engineers spend waiting for slow builds.

To start calculating the cost of a build, we need the the following concrete pieces of data:

• # engineers
• # builds per engineers per week
• # build duration in mins
• $ cost of a minute of an engineer’s time

For example, your team might have

• 10 engineers
• 25 builds a week each (each engineer doing 5 builds each week day)
• 30 minute build time
• $1 per minute for an engineer’s time ($125k average salary / 260 work days a year/ 8 hours a day / 60 minutes)

Next, a slightly trickier estimate. What percentage of time does a developer spend waiting on a build?

I debate this much more extensively below, but here I am going to make a very conservative estimate that 20% of the time a build takes to complete is wasted. 

So, with our example numbers, we have

• 10 engineers
• Each doing 25 builds a week = 250 builds
• With each build taking 30 mins = 7500 build minutes
• We are assuming engineers are waiting for 20% of that time = 1500 minutes waiting
• At a cost of $1 per minute = $1500

That means our slow build are costing us $1500 per week, or approximately $75,000 per year. And of course, if you assume 100% of build time is wasted, the numbers would be 5x ($7,500 per week, or approximately $375,000).

And for every minute you can trim off that build time, you could save $50-250$ per week, or $2,500-$12,500 per year.

You can see these simple calculation in this spreadsheet or see Palmer’s original and more comprehensive version here.

 

Should I prioritize speeding up the build?

 

If the numbers you calculate based on the above approach justify it, go to work on it straight away. Often, however, this work contends for engineering resources, just like any other piece of work. I recommend reading the cost of delay prioritization discussion in Palmer’s Cost of Delay article. Consider the level of effort required to do the work, and weigh it against the other items your team and Product Manager are considering.

Independent of the other project work you have going on however, longer build times are always a challenge. Slow builds essentially force a context switch and we have enough other things forcing context switches as it is (meetings, your boss, life) without forcing them for entirely avoidable reasons like slow build times.

This excellent Developer Productivity Engineering eBook (free from gradle.com) suggests that shorter build times increase productivity because there are fewer interruptions and that:

• Faster builds improve the creative flow
• The longer the build the more context switching
• Longer builds lead to more merge conflicts
• Longer builds are harder to debug

 

 

How slow is too slow?

 

Setting aside the monetary aspects of a slow build, is there a rule of thumb for how slow is too slow? After all, build time will never be zero, so what is a reasonable target?

 

10 minutes to build?

How long is too long for a build (industriallogic.com) suggests that 10 minutes is a good time to aim for. This is echoed in Martin Fowler’s Continuous Integration article, where he says “For most projects, however, the XP guideline of a ten minute build is perfectly within reason.”

That being said though, I have seen build times on large enterprise, monolithic projects take up to an hour. I can also attest to how painful that it though.

 

15 minutes to build and deploy to production?

The always interesting Charity Majors goes further and says that getting your code not just built, but deployed into customers hands in less than 15 mins should be your goal. In my humble opinion, that is an incredibly ambitious and lofty goal for many teams, but I love it none the less.

 

What percentage of time does a developer spend waiting on a build?

 

Above I assume what I believe to be a very conservative (low) estimate of 20%. But let’s debate that percentage of time an engineer wastes waiting for a build to complete further by starting with the extremes: All or nothing.

Does an engineer waste all of the time the build is running, staring at the build until it is passed and doing literally no other work? Maybe not. They might not do much productive work though (“Time to check Twitter!”). And there are certainly some people who would argue that all build time is wasted time, such as the aforementioned Palmer article, which states “there are valid questions as to the amount of downtime that developers actually incur when waiting for builds to complete” but goes ahead and assumes 100% of build time is wasted time. Still, I’m forever an optimist however, so let’s assume that while waiting for a build to complete, an engineer might go off and do some form of work, like answer emails or check slack. Let’s assume the % of wasted time is < 100%.

Similarly, does an engineer lose no time? Seamlessly switching context, and working completely productively right up until the build completes, when they again make a seamless switch back to their pre-build coding? Also probably not. Context switching is a known concentration killer. And they are certainly not likely to keep working on the same piece of code – they need to see if it builds first! Let’s assume the % of wasted time is > 0%.

So far, we’re assuming that engineers waste somewhere in between 0 and 100% of the time a build takes to complete. The “Quantifying the Costs of Builds” article states that you waste could be 80% of the time it takes for local builds and 20% for CI builds. That seems reasonable, and for the sake of simplicity, I am going to assume the lower 20% number, but bear in mind it could be up to 4 or 5 times higher that in reality (actually possibly even more if that twitter distraction lasts longer than the build time, but we have to draw the line somewhere for our estimates).

In summary, I am assuming that 20% of the time a build takes to complete is wasted, e,g., from the context switching, getting distracted, or checking the build status etc. You, however, can adjust between 0 and 100+% as you wish.

 

How do I get the build stats?

The calculations above require two build stats: build duration and number of builds.

 

Number of builds

You are likely dealing with 2 types of builds: local and CI.

CI builds should be easier to get hard data for. Tools like Gitlab Analytics, CircleCI Insights and Azure Pipeline reports can provide stats on the number of runs, duration, pass/fail rates etc, so you shouldn’t have to guess.

Local build times are much harder to get hard data for. Every time you locally run mvn install, gradlew build or whatever your poison of choice is, there is likely no central place where such runs are being tracked. So, you may need to estimate. You may want to base the estimate of your CI build stats, based on the premise that engineers usually run the build locally much more than via the CI build. Your estimates may vary but a factor of 2X to 10X may be appropriate, and will be affected by whether you are practicing CICD or not. CICD mandates frequent merges, and hence likely more CI builds. Long running local branches probably skew more to local builds.

 

Build duration

This should be the easier one. You can pull the duration from pretty much any build run using most build tools, including the CI tools discussed above. If the local builds time differ significantly from the CI build times, consider calculating costs separately, but if they are not drastically different, and average of the two may well suffice.

 

Sources and recommended reading

• Prioritizing with Cost of Delay (jpalmer.dev) by Jeff Palmer
• Quantifying the Costs of Builds (gradle.com) by Hans Dockter @ Gradle
• How long is too long for a build (industriallogic.com)
• Developer Productivity Engineering eBook (gradle.com)
• Continuous Integration (martinfowler.com) by Martin Fowler

 

Tags: build, ci, cicd, continuousdelivery, continuousdeployment, continuousintegration, devops, metrics