Shift Left

Defining the term shift left to mean testing earlier in the development cycle feels antiquated since waiting until development is “complete” before testing is a plain ol’ anti-pattern at this point.

A better definition could be testing earlier and more frequently. Writing tests each sprint, and running those tests with every commit.

Perhaps better still is thinking of shift left as a movement of testing to earlier in the pipeline. Favor unit tests, which typically run in the pipeline first and fast. Heavier-weight tests, such as UI based and end-to-end tests, which are typically harder to write, slower to run, and run later in the pipeline, do have a place but should be used sparingly.

Another way to look at this is that shift left means shifting down the testing pyramid.

 

 

First, the term shift left is most often used to refer to testing, but is increasingly being used in other disciplines such as security, devops and performance testing. In this article, to keep things simple, I am going to focus on its original usage in testing.

Testing earlier in the dev cycle

Shift left is often defined as the practice of moving important software considerations to earlier in the development cycle.

“Shift-left testing is an approach in which testing is performed earlier in the lifecycle i.e. moved left on the project timeline”

Shift left testing (wikipedia.org)

“With the traditional Waterfall model, testing is done just before releasing the product into production. This means that serious problems uncovered so late can cause major redesign and long delays. Shift Left Testing addresses this by involving testing teams early in the process.”

Shift Left (devopedia.org)

In the waterfall approach (as shown in the diagram below), shift left can mean thinking about testing at the coding, design, or even requirements phase, rather than after code has been written.

Waterfall Model (techtarget.com)

 

This “earlier in the development cycle” feels like an antiquated definition to me however. Isn’t that just common sense? Not many folks use waterfall anymore, and even if you do, it is difficult to fathom not testing until after development has been completed. Is the term shift left in that context even useful?

 

Testing earlier, and repeatedly

Waterfall, and the whole idea of a linear development cycle, has fallen out of fashion anyway. Agile is more common. Although the word “agile” in software development circles has become so diluted to have almost no meaning anymore. But let’s just focus on one of the Principles behind the Agile Manifesto:

Deliver working software frequently, weeks to months, with a preference to the shorter timescale.

Principles behind the Agile Manifesto (agilemanifesto.org)

So we are working on smaller iterations (e.g. sprints) and thinking about testing early and often in each of them. That feels better. But is it noteworthy? Not really! Hopefully you’re doing that already, and better still, even using TDD to write tests before the code. Again, is that really a new or useful perspective?

 

Testing earlier in the pipeline

So if considering quality and writing tests at much earlier stages of the dev cycle is already the norm, what is shift left?
How about: Shift left means moving tests earlier in the pipeline.

Not just writing tests earlier, but RUNNING tests earlier too, and run them frequently.

Shift-left testing is really about continuous testing. Automation is essential. This reduces human errors, increases test coverage and lets testers focus on more inspiring tasks.

Shift Left (devopedia.org)

And on a side note…

What is a deployment pipeline?

A deployment pipeline is a system of automated processes designed to quickly and accurately move code from version control to production.

What is a Deployment Pipeline? (pagerduty.com)

What is the order of a deployment pipeline?

Maybe:

• Commit
• Compile
• Static tests
• Units tests
• Integration Tests
• Deployment
• UI tests
• End to End tests
• Performance and Load tests
• Release

Or:

DevOps Pipeline (atlassian.com)

 

Shift down the testing pyramid

And perhaps another similar way to look at this is that shift left means shifting down the testing pyramid.

What is a Testing Pyramid?

A Test Pyramid is “a way of thinking about how different kinds of automated tests should be used to create a balanced portfolio.”

Test Pyramid (martinfowler.com)

The idea being that you have more tests at the “base” of the pyramid, in the form of low-level unit tests, than you do high-level UI tests, since those are both slower to run and more costly to write.

Instead of UI, Service and Unit tests, the same Pyramid can also be described using the terms End to End (E2E), Integration and unit tests, as Shreya Bose does here in this BrowserStack article:

 

Testing Pyramid : How to jumpstart Test Automation (browserstack.com)

I also like this definition from Kent Dodds, where he defines it as a testing Trophy, and includes static tests too:

The Testing Trophy (twitter.com)

See also The Testing Trophy and Testing Classifications (kentcdodds.com)

 

With the test pyramid defined, how does shift left fit in?

Another way to think about shift left is to move tests down the testing pyramid (since tests at the bottom of the pyramid tend to come first in the pipeline)

Shifting left, or shifting down the pyramid, would mean favoring unit tests over integration tests, and favoring integration tests over unit tests. For example, if you catch a bug with a UI test, can you write an integration test to catch it first? If you catch it with an integration test, can you write a unit test instead? Catch as many issues as you can as early in the pipeline as you can.

I’ve written before about how much more expensive it can be to fix a bug later, in production instead of development versus production (How much longer does a bug take to fix in production?), referencing this Finding and fixing regressions article from Facebook that suggests it takes 20x longer to fix a big in production than at peer review.

Catching tests higher in the pyramid is costly too. E2E tests, for example, are often difficult to write, difficult to run (since they require all your systems to be up), slow, inconsistent, generate false negatives, and are hard to diagnose. As anecdotal evidence, I have seen many QA teams that only require a certain % of E2E tests to be passing (e.g. 90%), since all passing was such a rare phenomenon that the release couldn’t be blocked waiting for it.

Finally, catching later in the pipeline costs too. A test failing at the end of a one-hour pipeline (e.g. in an E2E test) is always going to be less optimal than catching it at the start of the pipeline. Those costs can include Infrastructure costs, the cost of delay, and the time engineers spend waiting. See more at How much is your slow build costing you?

 

Wrapping up

Yes, shift left can mean test early and often in the project lifecycle, but it also means moving tests from the slow expensive (and hence infrequently run tests) to the fast, cheap tests that you are running on every commit.

You can even take the definition further. In his “We need to talk about testing” article, Dan North says that shifting left means considering not just testing but stakeholders early, often and continually, and thinking about how testing and testers fit into software development.

 

Sources and references

• Shift left (devopedia.org)
• The Testing Trophy and Testing Classifications (kentcdodds.com)
• Testing in the Twenties (tbray.org)
• On the Diverse And Fantastical Shapes of Testing (martinfowler.com)
• Unit, Integration and End-To-End Tests – Finding the Right Balance (codeahoy.com)
• Transitioning from conventional to shift-left testing (thoughtworks.com)
• What is a Deployment Pipeline? (pagerduty.com)
• We need to talk about testing (dannorth.net)

Tags: deliverypipeline, quality, shiftleft, Testing, unittesting