1. Define the affected workflow

Record the start time, sending platform, message purpose, receiving providers, and observed symptoms. “Password resets rejected by one provider” is more actionable than “email is broken.” Include a baseline from before the incident if available.

Assign a coordinator and a decision log. Keep business impact separate from the technical hypothesis so a busy team can agree what needs containment first.

2. Gather the smallest useful evidence pack

Collect sanitized status codes, trusted authentication observations, relevant sending IP/domain, correlation IDs, and timestamps. Avoid copying full customer messages or credentials into chat, tickets, dashboards, or public tools.

Use an authorized test account to reproduce the failure. Write down the exact message route and what the receiving system reported. If evidence is unavailable, record that gap instead of filling it with a likely explanation.

3. Choose a reversible containment step

Check recent changes to DNS, signing, sending platforms, lists, credentials, and suppression processing. If a rollback is appropriate, identify the known-good value, the owner, and the expected observation before acting.

Avoid increasing retry volume during a provider outage or repeatedly resending unwanted messages. Apply bounded retries and communicate partial/unavailable states honestly. Escalate shared infrastructure problems to the provider with the sanitized evidence pack.

4. Prove recovery and preserve the lesson

Define the recovery check before closing the incident: a fresh successful workflow, relevant receiver observations, and a return toward the agreed operational baseline. One passing test may be necessary without being sufficient.

Write a concise follow-up: cause, contributing conditions, successful containment, preventive action, and owner/date for that action. Update the sender inventory and runbook so the next responder does not have to reconstruct the same facts.

A good runbook is short enough to use during an incident and specific enough to prevent a risky guess.

Before you move on

  • Impact and evidence separated from hypotheses.
  • Customer content and credentials redacted.
  • Containment has an owner and rollback plan.
  • Recovery criteria agreed and checked.