Diagnosing Intermittent WebSocket Disconnects at Scale

As a senior network and application debugging expert, your task is to construct a comprehensive diagnostic playbook for a specific WebSocket issue.

**Role:** Senior Network and Application Debugging Expert.

**Context:**
Our WebSocket clients are experiencing consistent disconnects approximately every 55 seconds. This behavior points towards potential issues with network intermediaries rather than application-level errors. We need to investigate three primary hypotheses:
1.  **Proxy Idle-Timeout:** An upstream proxy (e.g., Nginx, HAProxy, AWS ALB) might be terminating connections due to an idle timeout that is shorter than the WebSocket's natural heartbeat or activity.
2.  **TCP Keepalive Misconfiguration:** The operating system's TCP keepalive settings on either the client or server, or an intermediary, might be misconfigured, leading to premature connection closure.
3.  **Load Balancer Stickiness/Session Affinity:** If a load balancer is in use, a lack of proper session stickiness could be routing subsequent requests from the same client to different backend servers, breaking the WebSocket connection.

Our WebSocket service is named `{{websocket_service_name}}`. We suspect `{{proxy_type}}` is acting as a proxy, and `{{load_balancer_type}}` is handling load balancing.

**Task:**
Generate a detailed debug playbook. The playbook must systematically address the symptom, list hypotheses, outline specific checks with relevant commands, suggest likely fixes, and describe verification steps.

**Constraints:**
*   The output must follow the exact structure: Symptom → Hypothesis List → Checks (with commands) → Likely Fixes → Verification.
*   Each 'Check' should include specific command-line tools or configuration inspection steps relevant to Linux environments or common proxy/load balancer types.
*   'Likely Fixes' should be actionable and configuration-focused.
*   'Verification' should describe how to confirm the fix.
*   Maintain a diagnostic, hypothesis-driven, and log-aware tone throughout.
*   Focus on the 55-second disconnect interval as the primary clue.

**Output:**

### Debug Playbook: WebSocket Disconnects at ~55 Seconds

**Symptom:**
WebSocket clients connected to `{{websocket_service_name}}` consistently disconnect after approximately 55 seconds of inactivity or low activity.

**Hypothesis List:**
1.  **Proxy Idle-Timeout:** An intermediary proxy (`{{proxy_type}}`) has an idle timeout configured near 55-60 seconds, terminating connections.
2.  **TCP Keepalive Misconfiguration:** Default TCP keepalive settings on the server OS or an intermediary are closing connections.
3.  **Load Balancer Stickiness:** The load balancer (`{{load_balancer_type}}`) is not maintaining session affinity, causing connection resets.

**Checks (with commands):**

**1. Proxy Idle-Timeout Investigation (`{{proxy_type}}`):**
*   **Check `{{proxy_type}}` Configuration:**
    *   **Nginx:** Inspect `nginx.conf` or relevant virtual host configurations for `proxy_read_timeout`, `proxy_send_timeout`, `keepalive_timeout`. Look for values around 60s. Example: `grep -r 'timeout' /etc/nginx/conf.d/ /etc/nginx/nginx.conf`
    *   **HAProxy:** Examine `haproxy.cfg` for `timeout client`, `timeout server`, `timeout connect`, `timeout http-request`. Example: `grep -r 'timeout' /etc/haproxy/haproxy.cfg`
    *   **AWS ALB:** Check the idle timeout setting for the target group associated with `{{websocket_service_name}}` in the AWS console or via AWS CLI: `aws elbv2 describe-target-groups --names {{websocket_service_name}}-target-group` (adjust target group name).
*   **Check Proxy Logs:** Review `{{proxy_type}}` access and error logs for connection termination messages or specific timeout errors around the 55-second mark. Example (Nginx): `tail -f /var/log/nginx/access.log /var/log/nginx/error.log | grep '504\|timeout'`

**2. TCP Keepalive Misconfiguration:**
*   **Check Server OS TCP Keepalive Settings:**
    *   Inspect `sysctl` parameters on the server hosting `{{websocket_service_name}}`:
        `sysctl net.ipv4.tcp_keepalive_time`
        `sysctl net.ipv4.tcp_keepalive_intvl`
        `sysctl net.ipv4.tcp_keepalive_probes`
    *   Look for `tcp_keepalive_time` values around 60 seconds. Default is often 7200s (2 hours), but can be overridden.
*   **Check Client-Side Keepalive (if applicable):** If client-side libraries allow, verify their keepalive or heartbeat settings.

**3. Load Balancer Stickiness (`{{load_balancer_type}}`):**
*   **Check `{{load_balancer_type}}` Configuration:**
    *   **AWS ALB:** Verify target group stickiness settings (e.g., 'Stickiness enabled' and duration). Ensure it's cookie-based and configured for a sufficient duration for WebSocket sessions.
    *   **Other LBs:** Consult documentation for `{{load_balancer_type}}` regarding session affinity or stickiness for WebSocket connections. Often requires IP-based or cookie-based stickiness.
*   **Network Trace (Optional but Recommended):** Use `tcpdump` or Wireshark to capture traffic between the client, load balancer, and server. Look for `FIN` or `RST` packets originating from an unexpected source or after the ~55s interval. Example: `sudo tcpdump -i any -nn port 80 or port 443 -s0 -w websocket_debug.pcap`

**Likely Fixes:**

**1. Proxy Idle-Timeout Adjustment:**
*   **Nginx:** Increase `proxy_read_timeout` and `proxy_send_timeout` to a value significantly higher than the expected idle period (e.g., 3600s for 1 hour). Also, ensure `keepalive_timeout` is adequate.
*   **HAProxy:** Increase `timeout client` and `timeout server` in the relevant frontend/backend sections to a higher value (e.g., `3600s`).
*   **AWS ALB:** Increase the 'Idle timeout' setting for the target group to a value like 3600 seconds (1 hour) or more.
*   **Implement WebSocket Heartbeats:** Configure the WebSocket application to send small heartbeat messages (ping/pong frames) more frequently than the suspected timeout (e.g., every 30 seconds).

**2. TCP Keepalive Configuration:**
*   **Server OS:** If `net.ipv4.tcp_keepalive_time` is set low, increase it to a higher value (e.g., 300 seconds or more) or revert to the default. Apply with `sudo sysctl -w net.ipv4.tcp_keepalive_time=300` and persist in `/etc/sysctl.conf`.
*   **Proxy/Load Balancer:** Some proxies/LBs have their own TCP keepalive settings that may need adjustment.

**3. Load Balancer Stickiness Configuration:**
*   **`{{load_balancer_type}}`:** Ensure session stickiness is enabled and correctly configured for WebSocket traffic. For HTTP/HTTPS load balancers, this often means cookie-based stickiness with a long duration. For pure TCP load balancers, IP-based stickiness might be the only option.

**Verification:**
1.  **Monitor Client Connections:** Observe client connections for an extended period (e.g., several hours) to confirm the absence of ~55-second disconnects.
2.  **Check Logs:** Review `{{websocket_service_name}}` application logs, `{{proxy_type}}` logs, and `{{load_balancer_type}}` logs for any connection termination events or errors after applying fixes.
3.  **Simulate Inactivity:** Intentionally leave a WebSocket client idle for longer than the previous disconnect interval (e.g., 2-3 minutes) to confirm the connection persists.
4.  **Network Trace (Post-Fix):** If `tcpdump` was used for initial diagnosis, run it again to confirm `FIN`/`RST` packets are no longer prematurely terminating connections.