Shaun Abram
Technology and Leadership Blog
Subscribe to RSS FeedBeginning with SRE
This post is an introduction into some basic SRE practices we have been implementing at my company recently.
I’ve written before on SRE, including on SRE resources, SLIs, SLOs and SLAs, and Creating an SRE team, but this is a more practical guide to getting started.
Tags: servicelevelagreements, sitereliabilityengineering, sla, sli, slo, sre, thesrebook
SRE Metrics
A very quick post on some of the most commonly used SRE metrics: The Four Golden Metrics, and RED & USE.
Tags: 4goldenmetrics, metrics, sitereliabilityengineering, sre
Creating an SRE team
If you wanted to build an SRE team at your company, how would you go about it? How would you structure it?
Tags: seekingsrebook, sitereliabilityengineering, sre, sreworkbook
SRE Resources
The following are a list of SRE resources I’m finding useful. I will update it as I find more. The good news is that most of the books (including all 3 of the Google SRE books) are available for free download at https://landing.google.com/sre/books.
Tags: seekingsrebook, sitereliabilityengineering, sre, srebooks, sreworkbook, thesrebook
eBook Summary: What Is SRE?
“What Is SRE? An Introduction to Site Reliability Engineering” (registration required but free), is an ebook by Kurt Andersen & Craig Sebenik, published by O’Reilly. The following is a summary (abridged copy and paste) of the parts I found most useful, with a few of my own notes. The original is about 9,000 words; this is about 2,000.
Tags: books, sitereliabilityengineering, sre, summary
SRE vs DevOps
I’m really enjoying the Seeking SRE book. Chapter 12 covers SRE vs DevOps; a community sourced compare and contrast type discussion.
My favorite description is from Thomas Limoncelli, who suggested that:
DevOps engineers focus on the SDLC pipeline with occasional responsibilities for production operations. SREs focus on production operations with occasional responsibilities for the SDLC pipeline.
Tags: books, devops, seekingsrebook, sitereliabilityengineering, sre
Book chapter summary: Postmortem Culture, from the SRE Book
I’m really enjoying reading the excellent “SRE Book“. Chapter 15 “Postmortem Culture: Learning from Failure” in particular, really struck a chord with me. The following is a slightly summarized version of it.
TLDR: Failures are inevitable, especially in distributed systems. To learn from them, document in Postmortems, avoiding blame, and share the newly gained learnings across your org.
Tags: blameless, postmortems, RCA, rootcauseanalysis, sitereliabilityengineering, sre, summary, thesrebook
Talk summary: SRE principles by Tori Wieldt @ AWS re:Invent 2018
I caught a talk by Tori Wieldt at the New Relic booth at AWS re:Invent on “SRE principles”. Even though it was a short talk in the expo hall, rather than a formal scheduled one, it had a ton of good SRE material.
Tags: aws, newrelic, reinvent, reinvent2018, sitereliabilityengineering, sre, summary, Testing
Summary: The Observability Crisis
The Observability Crisis is an article from Jaya Gupta & Ashu Garg from Foundation Capital, a Silicon Valley based venture capital (VC) firm investing in tech startups.
TLDR:
Companies in the first wave of the observability space (such as Splunk, AppDynamics, Datadog and New Relic) focused on solving data storage and analysis problems. However, with ever-increasing data volumes, costs increase and extracting actionable insights becomes more difficult.
Companies in the second wave (such as Honeycomb, Grafana and Cribl) helped organizations cope with complexity and cost by better supporting an increase in the volume, velocity, and the cardinality of data. However, the engineering overhead (such as SREs) to manage these tools, control costs, and triage alerts remains high.
The third wave could involve a shift from volume-based to value-based pricing, where you pay for observability tools based on their ability to accurately identify root causes and even prevent them from recurring. This is likely to involve using LLMs along with ML techniques that:
- Offer predictive maintenance by recognizing patterns across heterogeneous data sources
- Automatically generate incident playbooks
- Synthesize data from post-mortem reports
So, while I do like this article, my main gripe is that the article leaves it unclear whether any such “3rd wave” tools exist yet.
Read on for a summary of that article. The original is ~1600 words. This is ~600.
Tags: apm, devops, incidents, observability, RCA, reliability, rootcauseanalysis, sitereliabilityengineering, sre, summary
Blog post summary: Why we do incident drills
Why we do incident drills and why you should too is an article by Fína Mádrová, an SRE at Mews. This is a summary (600 words) of the original (1400 words).
Tags: incidents, sitereliabilityengineering, sre, summary
Why to avoid Mean Time to Recover (MTTR)
The 2022 Void Report came out in late 2022, It is a recommended read, and I previously summarized it here. This article focuses on one aspect of the report: why mean time to recover (MTTR) is not an appropriate metric for complex software systems.
The takeaways are:
- Do track time to recover (TTR) for each incident. It can be a useful exercise to think about when an incident started and stopped. That can help when calculating the cost of an incident.
- Don’t report those times in aggregate, such as MTTR. Systems fail in non-uniform ways and averaging numbers to represent their reliability (or the performance of the supporting teams) is likely to be misleading.
- Instead, use:
- Post-incident learning reviews to learn (and share!) everything you can from an incident
- SLOs to help align technical system metrics with business objectives
- Consider sociotechnical incident data too
Tags: accelerate, dora, incidents, mttr, reliability, sitereliabilityengineering, sre, voidreport
Summary: The SPACE of Developer Productivity
The SPACE of Developer Productivity is a 2021 paper by researchers at GitHub, University of Victoria, and Microsoft (including Dr Nicole Forsgren, co-author of Accelerate) that looks into ways to measure and predict productivity for both individuals and teams.
The following is a summary of the paper. The original is ~5400 words. This is ~2000.
Tags: developerproductivity, metrics, productivity, summary
2022 VOID Report Summary
The following is a summary of the 2022 VOID report (pdf can be found directly here)
The original is ~10,000 words. This is ~1500.
Tags: incidents, summary, voidreport
Book chapter summary: Managing Incidents
This is a slight abridged version of Chapter 14, “Managing Incidents, by Andrew Stribblehill from the excellent “SRE Book“. (Original is 2200 words, this is 1200)
Tags: sitereliabilityengineering, sre, summary, thesrebook
Development and delivery practices for team success
Most metrics for measuring developer productivity, such as lines of code or issues closed, are notoriously ineffective. But the research in the excellent State of Devops report shows that, rather than focusing on local metrics and individual developer performance, it is better to look at overall development and delivery practices. Specifically, there are metrics that predict and reflect a team’s ability to successfully deliver working software into production, including deployment frequency, and the mean time to restore service after an incident. This articles discusses why some metrics are useless, and takes a closer look at the recommendations in the 2019 State of Devops report.
Tags: accelerate, developerproductivity, devops, fourkeymetrics, stateofdevops