Blog post summary: Why we do incident drills

Why we do incident drills and why you should too is an article by Fína Mádrová, an SRE at Mews. This is a summary (600 words) of the original (1400 words).

Toggle

Intro

Bugs and issues can be reduced by good practices and architecture, but serious ones that rise to the level of an incident are inevitable. An incident handling process, including deliberate incident practice and drills, is essential to react quickly and overcome such critical events.

Past challenges

Past challenges of incident management at Mews included:

A manual, impractical postmortem process: A 19-point checklist with a lengthy postmortem process that required addressing every identified root cause, no matter how impractical the solution. Each proposed change had to be done via a rigid Request for Comments (RFC) process, discouraging small incremental changes.

Lack of training: Insufficient preparation of incident responders led to many deviations from the process, creating confusion and unnecessary stress.

Establishing a new process

We developed a new process that

• Guides towards a resolution by offering prompts for tasks that need human input (such as status updates), while automating those that don’t (like creating channels, cross-posting information, or adding relevant people)
• Uses incident.io to help us establish good practices and do the heavy lifting, rather than a homegrown solution
• Incorporated incident drills

Incident response drills

While we recognize that everyone has different learning preferences – whether through documentation, videos, or observing real incidents – we believe there is value in hands-on experimentation and addressing specific team needs and concerns.

It can be difficult to convince some people to participate in drills (“Don’t we have enough live incidents?”), but others are relieved to have someone to answer questions.

We want all responders to be familiar with the tools and processes at their disposal so they don’t need to search for documentation or reinvent solutions during incidents.

During these drills, nothing is actually broken. An experienced incident responder guides the team through the incident lifecycle, explaining what the tools can do, what happens automatically, and where input is needed, highlighting the importance of sharing good internal and external updates. A playful scenario can help people relax and engage more than a typical “Incident drill” or “NullReferenceException”. ChatGPT is useful for brainstorming new topics.

There should be enough time for discussion, and to allow people to try the steps for themselves.

Shadowing

It can be helpful to shadow an experienced incident responder during an active incident. When shadowing, try to determine if you could perform the same steps yourself, or even step in to take some of the tasks with guidance. For example, you might discover you lack some needed system access.

Afterwards, talk to the responders and ask them to walk you through what they did so you can learn.

Team specific runbooks

Knowing the processes is good, but resolution is king. To that end, consider writing a team runbook of typical remediation steps your team takes, like

• restarting a service,
• performing a failover,
• rolling back a deployment, or
• recreating a corrupted cloud resource

Some may seem trivial to the author, but such a checklist or tutorial can be a godsend to less knowledgeable incident responders during a stressful time.
Better still, automate the steps too.

Chaos engineering

Break something and observe:

• Time to detect? (Is there sufficient observability and alerting?)
• How does the response team react?
• Can the system recover on its own?

Conclusion

Incident drills are essential for preparing teams to handle unexpected issues effectively. By establishing an easy-to-follow process, conducting basic incident response drills, shadowing experienced responders, and creating team-specific runbooks, teams can better equip themselves to manage incidents confidently and efficiently.

Tags: incidents, sitereliabilityengineering, sre, summary