SRE Incident Response Template: Roles, Timelines, Communications, and Postmortems

Overview

Incident response is the coordinated process used to detect, assess, contain, recover from, and learn from a service disruption or security-relevant event. The goal is not to eliminate every incident. The goal is to reduce confusion, limit impact, restore dependable service, and make the next response more effective.

A good incident response runbook should answer five questions quickly:

• What is happening, and how severe is it?
• Who is coordinating the response?
• Which technical actions are safe to take now?
• Who needs an update, and when?
• What evidence will show that recovery is complete?

Keep the playbook close to the systems your responders already use, such as the alerting platform, chat workspace, ticketing system, or internal developer platform. Link it from service documentation and deployment procedures. For production changes, pair this process with a Kubernetes deployment checklist or an equivalent rollback guide so responders can move from diagnosis to a controlled recovery path.

Severity definitions should be simple enough to apply consistently. For example, a critical incident may involve a broad loss of a customer-facing service, a serious data integrity concern, or a security event requiring immediate escalation. A high-severity incident may affect an important function or a significant group of users but have a viable workaround. Lower severities can cover limited degradation, isolated failures, or issues that do not require continuous coordination. Define the exact thresholds for your organization, then review whether responders classify similar events in similar ways.

What to track

Roles and ownership

Assign roles explicitly at the start of an incident. One person may hold more than one role in a small team, but naming the responsibilities prevents important work from disappearing.

• Incident commander: Owns coordination, sets priorities, makes or delegates decisions, and declares when the incident is closed.
• Technical lead: Directs investigation, proposes mitigations, and keeps technical notes factual and time ordered.
• Communications lead: Publishes internal, customer-facing, or partner updates using approved language and a predictable schedule.
• Scribe: Records key observations, actions, decisions, timestamps, and links to evidence.
• Subject-matter experts: Investigate specific services, dependencies, infrastructure layers, or security concerns.
• Business or customer liaison: Helps assess user impact and coordinates with support, account, legal, or leadership teams when needed.

The incident commander should not become the primary person making every technical change. Separating coordination from investigation helps preserve context and gives the commander room to monitor risk, dependencies, and communication.

Timeline and operational facts

Track the first alert time, first human acknowledgement, incident declaration, responder assignments, major findings, mitigation attempts, rollback or failover actions, recovery indicators, and closure time. Record time zones consistently. Use links to dashboards, logs, traces, deployment records, tickets, and relevant chat threads rather than copying large volumes of raw data into the incident record.

Technical triage

Start with a narrow working hypothesis and test it against observable evidence. Check the scope of the failure, affected services, recent changes, dependency health, error rates, latency, saturation, capacity, authentication paths, and unusual traffic. Avoid making several unrelated changes at once; doing so can make the cause harder to identify and can increase the blast radius.

For Kubernetes-based systems, useful checks may include workload status, recent rollout history, pod events, service endpoints, ingress behavior, resource pressure, and dependency connectivity. The Kubernetes deployment troubleshooting checklist can provide a structured starting point. If telemetry is inconsistent, review your collection path and sampling decisions using patterns from the OpenTelemetry Collector configuration guide.

Communications

Every update should state what is known, what is not known, what is being done, and when the next update will arrive. Avoid speculation presented as fact. A useful internal update looks like this:

• Impact: Describe the affected service, users, region, or function.
• Current status: Investigating, mitigating, monitoring, or recovered.
• Evidence: Link to the relevant dashboard, alert, or incident record.
• Action: State the next technical or coordination step.
• Next update: Provide a specific time or trigger.

Customer communication should focus on observable impact, work in progress, and available workarounds. Do not include internal blame, unverified root causes, or sensitive implementation details. Security-related incidents may require a separate escalation path and controlled handling of evidence.

Recovery validation

Recovery is more than a green deployment or a falling error rate. Confirm that key user journeys work, background processing has caught up, queues are draining safely, dependent services are healthy, and monitoring is stable across a reasonable observation period. Check whether the mitigation created secondary effects, such as delayed jobs, elevated costs, stale caches, or reduced capacity.

Cadence and checkpoints

Use a lightweight cadence during the incident so the process does not depend on memory.

1. First 10 minutes: Acknowledge the alert, declare or reject the incident, assign an incident commander, identify initial impact, and open the shared record.
2. Next 15 to 30 minutes: Confirm severity, assign technical and communications roles, establish the working hypothesis, and choose a low-risk mitigation or rollback path.
3. Every scheduled interval: Publish an update even when there is no major change. State what was checked and what remains uncertain.
4. At mitigation: Record the exact change, confirm that impact is declining, and keep monitoring rather than closing immediately.
5. At closure: Confirm recovery criteria, communicate resolution, preserve evidence, and assign follow-up owners.

After closure, hold a review while the timeline and decisions are still accessible. The review should be blameless: examine system conditions, process gaps, unclear ownership, missing telemetry, unsafe defaults, and barriers that made the right action difficult. A strong observability tools comparison can also help teams evaluate whether their current signals support timely detection and diagnosis.

For delivery-related incidents, compare the incident timeline with the deployment record and CI/CD controls. Reusable workflows, test gates, security checks, and rollback steps should be easy to inspect. The GitHub Actions CI/CD pipeline templates provide useful areas to review when release automation is part of the failure path.

How to interpret changes

Track recurring indicators across incidents rather than judging the process from a single event. Useful measures include time to acknowledge, time to assign an incident commander, time to mitigation, time to recovery, number of handoffs, number of rollback attempts, repeat incidents by service, and the percentage of postmortem actions completed on schedule. These measures are most useful as signals for discussion, not as targets that encourage responders to close incidents prematurely.

Interpret trends in context. A rise in incident count may reflect improved detection, a wider service footprint, or a genuine reliability regression. A shorter recovery time may be positive, but investigate whether teams are closing incidents before delayed effects are understood. Repeated incidents with different symptoms may share a common condition, such as weak dependency isolation, insufficient capacity planning, unclear ownership, or a deployment process that makes safe rollback difficult.

Review the quality of the evidence as well. If responders regularly debate which dashboard is authoritative, cannot correlate logs with deployments, or lack visibility into downstream dependencies, the process needs an observability improvement rather than another escalation rule. If incidents repeatedly involve access changes or exposed interfaces, include the relevant cloud security controls, IAM review, and audit evidence in the follow-up work.

Prioritize postmortem actions by risk reduction and feasibility. Examples include adding a missing alert, documenting a rollback, limiting a permission, introducing a canary, improving a readiness check, testing a backup restore, or defining an owner for an unmanaged dependency. Each action should have one accountable owner, a due date, and a verification method.

When to revisit

Review this incident response template at least quarterly and after any major incident, organizational change, architecture change, or migration to a new observability, deployment, or communication tool. Revisit it sooner when responders report that a role is unclear, an escalation path is stale, a dashboard is unavailable, or a recovery step depends on one person’s undocumented knowledge.

Use a recurring checkpoint to ask:

• Do severity levels still match business and technical impact?
• Can the on-call responder declare an incident and find the right contacts?
• Are escalation paths current for services, dependencies, security, and customer support?
• Do runbooks describe safe rollback, failover, and recovery validation steps?
• Are alerts actionable, correlated, and linked to useful evidence?
• Are open postmortem actions reducing repeat risk?
• Have recent platform or infrastructure changes introduced new failure modes?

Make one practical improvement during each review rather than rewriting the entire playbook without testing it. Run a short tabletop exercise for a realistic scenario, such as a failed deployment, unavailable dependency, certificate problem, or regional degradation. Assign roles, walk through the first 30 minutes, and note where people hesitate. Then update the runbook, dashboards, access paths, or communication templates that caused the hesitation.

Finally, keep the template discoverable and versioned. Link related infrastructure and recovery documentation, including Kubernetes backup guidance where data recovery is relevant. A response process becomes reliable when teams practice it, measure its friction, and revise it as systems and responsibilities change.