Shaun Abram
Technology and Leadership Blog
Subscribe to RSS FeedSummary: The Observability Crisis
The Observability Crisis is an article from Jaya Gupta & Ashu Garg from Foundation Capital, a Silicon Valley based venture capital (VC) firm investing in tech startups.
TLDR:
Companies in the first wave of the observability space (such as Splunk, AppDynamics, Datadog and New Relic) focused on solving data storage and analysis problems. However, with ever-increasing data volumes, costs increase and extracting actionable insights becomes more difficult.
Companies in the second wave (such as Honeycomb, Grafana and Cribl) helped organizations cope with complexity and cost by better supporting an increase in the volume, velocity, and the cardinality of data. However, the engineering overhead (such as SREs) to manage these tools, control costs, and triage alerts remains high.
The third wave could involve a shift from volume-based to value-based pricing, where you pay for observability tools based on their ability to accurately identify root causes and even prevent them from recurring. This is likely to involve using LLMs along with ML techniques that:
- Offer predictive maintenance by recognizing patterns across heterogeneous data sources
- Automatically generate incident playbooks
- Synthesize data from post-mortem reports
So, while I do like this article, my main gripe is that the article leaves it unclear whether any such “3rd wave” tools exist yet.
Read on for a summary of that article. The original is ~1600 words. This is ~600.
Tags: apm, devops, incidents, observability, RCA, reliability, rootcauseanalysis, sitereliabilityengineering, sre, summary
Blog post summary: Why we do incident drills
Why we do incident drills and why you should too is an article by Fína Mádrová, an SRE at Mews. This is a summary (600 words) of the original (1400 words).
Tags: incidents, sitereliabilityengineering, sre, summary
Why to avoid Mean Time to Recover (MTTR)
The 2022 Void Report came out in late 2022, It is a recommended read, and I previously summarized it here. This article focuses on one aspect of the report: why mean time to recover (MTTR) is not an appropriate metric for complex software systems.
The takeaways are:
- Do track time to recover (TTR) for each incident. It can be a useful exercise to think about when an incident started and stopped. That can help when calculating the cost of an incident.
- Don’t report those times in aggregate, such as MTTR. Systems fail in non-uniform ways and averaging numbers to represent their reliability (or the performance of the supporting teams) is likely to be misleading.
- Instead, use:
- Post-incident learning reviews to learn (and share!) everything you can from an incident
- SLOs to help align technical system metrics with business objectives
- Consider sociotechnical incident data too
Tags: accelerate, dora, incidents, mttr, reliability, sitereliabilityengineering, sre, voidreport
Book chapter summary: Managing Incidents
This is a slight abridged version of Chapter 14, “Managing Incidents, by Andrew Stribblehill from the excellent “SRE Book“. (Original is 2200 words, this is 1200)
Tags: sitereliabilityengineering, sre, summary, thesrebook
Beginning with SRE
This post is an introduction into some basic SRE practices we have been implementing at my company recently.
I’ve written before on SRE, including on SRE resources, SLIs, SLOs and SLAs, and Creating an SRE team, but this is a more practical guide to getting started.
Tags: servicelevelagreements, sitereliabilityengineering, sla, sli, slo, sre, thesrebook
SRE Metrics
A very quick post on some of the most commonly used SRE metrics: The Four Golden Metrics, and RED & USE.
Tags: 4goldenmetrics, metrics, sitereliabilityengineering, sre
Creating an SRE team
If you wanted to build an SRE team at your company, how would you go about it? How would you structure it?
Tags: seekingsrebook, sitereliabilityengineering, sre, sreworkbook
SRE Resources
The following are a list of SRE resources I’m finding useful. I will update it as I find more. The good news is that most of the books (including all 3 of the Google SRE books) are available for free download at https://landing.google.com/sre/books.
Tags: seekingsrebook, sitereliabilityengineering, sre, srebooks, sreworkbook, thesrebook
eBook Summary: What Is SRE?
“What Is SRE? An Introduction to Site Reliability Engineering” (registration required but free), is an ebook by Kurt Andersen & Craig Sebenik, published by O’Reilly. The following is a summary (abridged copy and paste) of the parts I found most useful, with a few of my own notes. The original is about 9,000 words; this is about 2,000.
Tags: books, sitereliabilityengineering, sre, summary
Is Apdex useful?
I’ve been trying to figure out what SLOs to define for some services recently, and wondering if Apdex is a useful metric. (See my previous post on the difference between SLIs, SLOs and SLAs)
Tags: apdex, applicationperformanceindex, sitereliabilityengineering, slo, SLOs, sre
SLI, SLO and SLA
What are SLIs, SLOs and SLAs?
Service Level Indicators (SLIs) are metrics that you choose to measure the health and performance of your services. Service Level Objectives (SLOs) are the desired target for those indicators. Service Level Agreements (SLAs) build on this and include the consequences of not meeting those targets. All are fundamental to Site Reliability Engineering.
In this post, I’ll try to explain each in more detail, how they relate to each other, and some examples of each.
Tags: seekingsrebook, servicelevelagreements, sitereliabilityengineering, sla, SLAs, sli, SLIs, slo, SLOs, sre, thesrebook
SRE vs DevOps
I’m really enjoying the Seeking SRE book. Chapter 12 covers SRE vs DevOps; a community sourced compare and contrast type discussion.
My favorite description is from Thomas Limoncelli, who suggested that:
DevOps engineers focus on the SDLC pipeline with occasional responsibilities for production operations. SREs focus on production operations with occasional responsibilities for the SDLC pipeline.
Tags: books, devops, seekingsrebook, sitereliabilityengineering, sre
Book chapter summary: Postmortem Culture, from the SRE Book
I’m really enjoying reading the excellent “SRE Book“. Chapter 15 “Postmortem Culture: Learning from Failure” in particular, really struck a chord with me. The following is a slightly summarized version of it.
TLDR: Failures are inevitable, especially in distributed systems. To learn from them, document in Postmortems, avoiding blame, and share the newly gained learnings across your org.
Tags: blameless, postmortems, RCA, rootcauseanalysis, sitereliabilityengineering, sre, summary, thesrebook
Talk summary: SRE principles by Tori Wieldt @ AWS re:Invent 2018
I caught a talk by Tori Wieldt at the New Relic booth at AWS re:Invent on “SRE principles”. Even though it was a short talk in the expo hall, rather than a formal scheduled one, it had a ton of good SRE material.
Tags: aws, newrelic, reinvent, reinvent2018, sitereliabilityengineering, sre, summary, Testing