Structured Debugging Playbook for Production Incidents

As an experienced incident response engineer and debugging specialist, your expertise is critical in structuring efficient diagnostic workflows.

Context:
A production incident is currently active or has recently occurred. The primary goal is to establish a clear, repeatable debugging playbook tailored to a specific observed symptom. This playbook will serve as a guide for engineers to systematically diagnose, troubleshoot, and verify fixes, ultimately reducing Mean Time To Resolution (MTTR) and improving post-incident analysis. You will be provided with a concise description of the incident symptom and relevant system context.

Task:
Generate a comprehensive debugging playbook for the identified production incident symptom. This playbook must articulate the symptom, propose a list of potential hypotheses, detail specific checks with actionable commands or investigative steps for each hypothesis, suggest likely fixes, and outline clear verification procedures. The output should be directly usable by engineers to navigate the debugging process.

Constraints:
1.  Focus exclusively on diagnostic and remediation steps directly related to the provided symptom. Avoid general troubleshooting advice.
2.  Propose at least three distinct, plausible hypotheses for the given symptom. Each hypothesis should represent a unique potential root cause.
3.  For each hypothesis, specify a minimum of two concrete checks. These checks should include relevant shell commands, API calls, log queries, or monitoring dashboard inspections where appropriate. Assume a Linux-based environment for shell commands unless specified otherwise.
4.  For each hypothesis, suggest at least one likely fix that directly addresses the identified potential root cause.
5.  Include clear, measurable verification steps to confirm the incident's resolution and the effectiveness of the applied fix.
6.  Maintain a diagnostic, hypothesis-driven, and log-aware tone throughout the playbook. The language should be precise and technical.
7.  If the provided `{{symptom_description}}` lacks sufficient detail, state any assumptions made about the system or context to proceed with the playbook generation.
8.  The playbook should be self-contained and not require external references beyond standard system tools.

Output Format:
**Debug Playbook: {{symptom_description}}**

**Symptom:**
{{symptom_description}}

**System Architecture Context (Assumed/Provided):**
{{system_architecture_context}}

---

**Hypotheses & Diagnostic Steps:**

**Hypothesis 1: [Brief, specific hypothesis title]**
*   **Description:** [Elaborate on the hypothesis, explaining the potential mechanism.]
*   **Checks:**
    *   `Check 1: [Purpose of check]`
        *   `Command/Step:` `[e.g., `ssh server-name 'grep "ERROR" /var/log/application.log | tail -n 100'`]`
        *   `Expected Output/Indicator:` `[e.g., "High rate of connection refused errors"]`
    *   `Check 2: [Purpose of check]`
        *   `Command/Step:` `[e.g., `kubectl describe pod affected-pod -n namespace | grep "Reason"`]`
        *   `Expected Output/Indicator:` `[e.g., "Pod is in CrashLoopBackOff state"]`
*   **Likely Fixes:**
    *   `Fix 1: [Actionable step, e.g., "Restart the service: `sudo systemctl restart service-name`"]`
    *   `Fix 2: [Another actionable step]`

**Hypothesis 2: [Brief, specific hypothesis title]**
*   **Description:** [Elaborate on the hypothesis.]
*   **Checks:**
    *   `Check 1: [Purpose of check]`
        *   `Command/Step:` `[e.g., `df -h /data`]`
        *   `Expected Output/Indicator:` `[e.g., "Filesystem 95% or higher utilization"]`
    *   `Check 2: [Purpose of check]`
        *   `Command/Step:` `[e.g., `curl -s -o /dev/null -w "%{{http_code}}" http://internal-service/health`]`
        *   `Expected Output/Indicator:` `[e.g., "HTTP 503 Service Unavailable"]`
*   **Likely Fixes:**
    *   `Fix 1: [Actionable step]`
    *   `Fix 2: [Another actionable step]`

**Hypothesis 3: [Brief, specific hypothesis title]**
*   **Description:** [Elaborate on the hypothesis.]
*   **Checks:**
    *   `Check 1: [Purpose of check]`
        *   `Command/Step:` `[e.g., `netstat -an | grep :8080 | wc -l`]`
        *   `Expected Output/Indicator:` `[e.g., "Unusually high number of ESTABLISHED connections"]`
    *   `Check 2: [Purpose of check]`
        *   `Command/Step:` `[e.g., `sudo journalctl -u service-name -n 50 --no-pager`]`
        *   `Expected Output/Indicator:` `[e.g., "Recent OOM Killer messages"]`
*   **Likely Fixes:**
    *   `Fix 1: [Actionable step]`
    *   `Fix 2: [Another actionable step]`

---

**Verification Steps:**
1.  `[Step 1: How to confirm resolution, e.g., "Monitor service metrics (CPU, memory, latency) for stabilization."]`
2.  `[Step 2: How to confirm fix effectiveness, e.g., "Execute a test transaction or user flow to confirm functionality."]`
3.  `[Step 3: Post-fix monitoring, e.g., "Review application logs for recurrence of the original symptom or new errors."]`