Investigating Production Incidents Skill Reference

Overview

investigating-production-incidents helps a coding agent turn a production incident into a durable investigation folder in the repo. It uses the same live query methodology as debugging-production-errors, then adds a durable, committed handoff folder so the diagnosis, evidence, and summary survive the conversation for whoever ships the fix or revisits it later.

Skill contract

  • It reads logs only and never changes log data or code.
  • It opens investigations/<YYYY-MM-DD>-<slug>/ at the repo root before it queries anything.
  • It writes only the evidence that supports the diagnosis into evidence/, one file per distinct piece, instead of a full dump.
  • It redacts obvious credential and PII patterns such as API keys, bearer tokens, emails, and phone numbers before it writes evidence to disk.
  • It uses the same query methodology as debugging-production-errors, but adds folder creation, evidence capture, and a durable INVESTIGATION.md.
  • The folder is committed by default, so the handoff survives the conversation.
  • If no log connection exists, your agent points to Logs settings and stops the investigation.

Workflow

  1. Create the investigation folder. Your agent opens investigations/<YYYY-MM-DD>-<slug>/ at the repo root before any query runs.
  2. Run the same live query workflow and save evidence as it appears. Your agent writes only the evidence that supports the diagnosis into evidence/, one file per distinct finding.
  3. Map the findings back to source or deploy context. Your agent ties the saved evidence to the relevant code path, deploy, or version change when one exists.
  4. Write INVESTIGATION.md. Your agent records the incident in a durable handoff file at the folder root.

The folder contains:

  • investigations/<YYYY-MM-DD>-<slug>/ at the repo root
  • evidence/ for redacted supporting evidence
  • INVESTIGATION.md for the handoff write-up

INVESTIGATION.md uses these sections:

  • Summary
  • Timeline
  • Evidence
  • Root Cause
  • Deploy Correlation
  • Fix Recommendation

Deploy Correlation can stay out of the file when no version change lines up with the spike onset.

See also