Coding DebuggingIntermediate30 minSaves 30 minutes

Debugging OOMKilled Docker Containers: A Systematic Playbook

Q: Does this playbook apply to Kubernetes environments?

Yes, many concepts and checks translate directly. Use `kubectl top pod`, `kubectl describe pod`, and examine pod/container logs for OOMKilled events. The underlying cgroup mechanisms are similar.

Q: What if the container is very short-lived after being OOMKilled, making `docker exec` difficult?

For short-lived OOMs, focus on `docker logs` immediately after termination, `dmesg` on the host, and any pre-OOM metrics captured by `docker stats` or monitoring systems.

Q: My application isn't Java; how do I analyze application-specific memory?

Adapt step 5. For Node.js, consider `heapdump` or built-in profilers. For Python, use `memory_profiler`. The `ps aux` command inside the container remains universally useful for process-level insights.

Engineers facing intermittent OOMKilled errors in containerized services can diagnose and resolve memory issues using a structured, hypothesis-driven approach.

This playbook helps engineers systematically debug Docker containers terminated by OOMKilled errors. It guides through identifying root causes, from cgroup limits and process memory views to heap sizing, providing commands and remediation steps. Target specific memory leaks efficiently.

ChatGPT Gemini Claude

READY-TO-USE PROMPT

Copy Prompt

prompt.txt

Role: You are a senior DevOps engineer specializing in containerized application performance and reliability.

Context: An application running in a Docker container is experiencing intermittent `OOMKilled` terminations. The issue occurs randomly, making it difficult to pinpoint the exact cause. We suspect a memory leak or incorrect resource allocation. The goal is to generate a comprehensive debugging playbook to systematically identify and resolve the root cause.

Task: Develop a detailed debugging playbook for `OOMKilled` Docker containers. Structure the playbook with the following sections: Symptom, Hypothesis List, Checks (with commands), Likely Fixes, and Verification Steps. The playbook should cover common scenarios like cgroup memory limits, process-level memory consumption, and application-specific heap sizing.

Constraints:
*   Provide specific Linux commands for memory inspection within and outside the container.
*   Include guidance on interpreting command outputs.
*   Suggest remediation steps that address both infrastructure (Docker/Kubernetes) and application-level concerns.
*   The playbook must be actionable and provide clear next steps for an engineer.
*   Assume the container is running a `{{application_type}}` application, and its name is `{{container_name}}`.
*   Focus on practical, hands-on diagnostic steps.

Output:
Present the debugging playbook in a clear, step-by-step format:

### Debug Playbook: Container OOMKilled

**Symptom:**
*   Container `{{container_name}}` is being terminated with `OOMKilled` status in Docker logs or Kubernetes events.
*   Terminations are intermittent and not consistently tied to specific load patterns.

**Hypothesis List:**
1.  **Cgroup Memory Limit Exceeded:** The container's allocated memory (cgroup limit) is insufficient for its workload.
2.  **Application Memory Leak/Spike:** The application itself has a memory leak or a transient memory spike that exceeds its available resources.
3.  **JVM/Runtime Heap Misconfiguration:** For applications using runtimes like JVM, the heap settings are not optimized for the container's memory limit.
4.  **Sidecar/Helper Process Memory:** Other processes within the container (e.g., agents, sidecars) are consuming unexpected memory.
5.  **Host-Level OOM Killer:** The host system itself is under memory pressure, leading to the host's OOM killer terminating processes, including Docker daemon or containers.

**Checks (with Commands):**

**1. Review Container Logs for OOMKilled Event Details:**
    *   `docker logs {{container_name}}`
    *   `kubectl describe pod <pod-name> -n <namespace>` (if Kubernetes)
    *   *Interpretation:* Look for `OOMKilled` messages, exit codes (e.g., 137), and any preceding memory-related warnings from the application.

**2. Check Docker Cgroup Memory Limits:**
    *   `docker inspect {{container_name}} | grep -i 'Memory'`
    *   `cat /sys/fs/cgroup/memory/docker/<container_id>/memory.limit_in_bytes` (replace `<container_id>` with actual ID from `docker ps`)
    *   *Interpretation:* Compare `Memory` (limit) with `MemoryUsage` (current usage). If usage is consistently near the limit before OOM, the limit is too low.

**3. Monitor Container Memory Usage Over Time:**
    *   `docker stats {{container_name}} --no-stream` (for snapshot)
    *   `docker stats {{container_name}}` (for real-time monitoring)
    *   `kubectl top pod <pod-name> -n <namespace> --containers` (if Kubernetes)
    *   *Interpretation:* Observe memory trends. Is it a gradual increase (leak) or a sudden spike? Note peak usage before OOM.

**4. Inspect Process-Level Memory Inside the Container:**
    *   `docker exec -it {{container_name}} bash` (or sh)
    *   Once inside: `ps aux --sort -rss`
    *   Once inside: `top` (then press `M` for memory sort)
    *   *Interpretation:* Identify which specific process within the container is consuming the most memory. This helps differentiate between the main application and sidecars.

**5. Analyze Application-Specific Memory (e.g., JVM Heap):**
    *   If `{{application_type}}` is Java, enable GC logging: `-Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps`
    *   Use `jmap` or `jstat` for heap analysis if possible (requires JDK inside container or remote JMX access).
    *   *Interpretation:* Look for full GC cycles, long pause times, and heap exhaustion events. Compare `Xmx` settings to container memory limits.

**6. Check Host System Memory:**
    *   `free -h`
    *   `dmesg | grep -i oom`
    *   *Interpretation:* Ensure the host itself isn't running out of memory, which could trigger its own OOM killer.

**Likely Fixes:**

1.  **Increase Container Memory Limit:** If cgroup limits are too restrictive, increase `memory` and `memory-swap` in Docker run commands or Kubernetes resource limits (`limits.memory`). Start with a conservative increase and monitor.
2.  **Optimize Application Memory Usage:**
    *   Address identified memory leaks in the `{{application_type}}` code.
    *   Optimize data structures or algorithms.
    *   For Java, tune JVM heap (`-Xmx`, `-Xms`) and garbage collector settings. Ensure `Xmx` is less than the container's cgroup memory limit, accounting for non-heap memory.
3.  **Adjust Sidecar/Helper Process Resource Allocation:** If other processes are memory hogs, reconfigure them or move them to separate containers.
4.  **Scale Out/Distribute Load:** If the workload genuinely requires more memory than a single container can efficiently handle, consider horizontal scaling or sharding.
5.  **Upgrade Host Resources:** If host-level OOM is occurring, the underlying server may need more RAM.

**Verification Steps:**
1.  **Deploy Fix and Monitor:** Apply the chosen fix and redeploy the container.
2.  **Observe Memory Metrics:** Continuously monitor `docker stats` or `kubectl top` for the `{{container_name}}` container.
3.  **Check Logs for OOMKilled Events:** Verify that no new `OOMKilled` events occur over a sustained period under typical load.
4.  **Load Testing:** Conduct load tests to ensure the fix holds under peak conditions.

Estimated results

DifficultyIntermediate

Setup time30 min

Time saved30 minutes

Best modelsChatGPT, Gemini, Claude

Best audienceSoftware Development, Cloud Computing

Editor's note

Why this prompt matters

Engineers frequently encounter OOMKilled terminations in Docker containers, a problem that often manifests intermittently and without clear patterns. This makes root cause analysis particularly challenging, as the issue can stem from various sources ranging from incorrect resource allocations at the container orchestration level to subtle memory leaks within the application code itself. This playbook provides a structured, diagnostic framework for addressing such events.

It is designed for DevOps engineers, site reliability engineers, and software developers responsible for operating containerized services. The workflow guides users through a systematic process of forming hypotheses, executing specific diagnostic checks, and identifying targeted remediation strategies. By following this method, teams can move beyond guesswork, reducing the time spent on debugging and restoring service stability more quickly. Reach for this playbook whenever a containerized application unexpectedly terminates due to out-of-memory errors, especially when the occurrences are sporadic.

Anatomy

Prompt engineering breakdown

Role

You are a senior DevOps engineer specializing in containerized application performance and reliability.

Context

An application running in a Docker container is experiencing intermittent `OOMKilled` terminations. The issue occurs randomly, making it difficult to pinpoint the exact cause. We suspect a memory leak or incorrect resource allocation. The goal is to generate a comprehensive debugging playbook to systematically identify and resolve the root cause.

Goal

Develop a detailed debugging playbook for `OOMKilled` Docker containers. Structure the playbook with the following sections: Symptom, Hypothesis List, Checks (with commands), Likely Fixes, and Verification Steps. The playbook should cover common scenarios like cgroup memory limits, process-level memory consumption, and application-specific heap sizing.

Constraints

Provide specific Linux commands for memory inspection within and outside the container. Include guidance on interpreting command outputs. Suggest remediation steps that address both infrastructure (Docker/Kubernetes) and application-level concerns. The playbook must be actionable and provide clear next steps for an engineer. Assume the container is running a `{{application_type}}` application, and its name is `{{container_name}}`. Focus on practical, hands-on diagnostic steps.

Output format

Present the debugging playbook in a clear, step-by-step format with specific headings: Symptom, Hypothesis List, Checks (with Commands), Likely Fixes, and Verification Steps.

Why this structure works

Role priming establishes the expertise required to deliver reliable diagnostic advice. Explicit constraints on including specific Linux commands and interpreting their output ensure the playbook is practical and immediately useful. The structured output format guarantees a consistent, actionable guide, enabling engineers to systematically follow the diagnostic workflow.

Pick your version

Prompt variations

BeginnerWorks with any model

For engineers new to Docker debugging or those needing a simplified, high-level overview of OOMKilled issues with basic checks.

prompt.txt

Role: You are a helpful guide for new engineers learning Docker. Context: Your Docker container, named `{{container_name}}`, keeps stopping unexpectedly with an 'OOMKilled' message. This means it ran out of memory. We need simple steps to find out why. Task: Create a simple checklist to debug why `{{container_name}}` is running out of memory. Explain what to look for and basic commands to check. Constraints: Use easy-to-understand language. Focus on common reasons like the container needing more memory or the app `{{application_type}}` using too much. Provide basic commands for checking memory. Output: ### Basic OOMKilled Debug Checklist **What's Happening:** * Your container `{{container_name}}` is crashing with 'OOMKilled'. **Simple Checks:** 1. **Check Container's Memory Limit:** Is the Docker container allowed enough memory? Use `docker inspect {{container_name}}` and look for 'Memory' settings. 2. **See Container's Memory Use:** How much memory is `{{container_name}}` actually using? Run `docker stats {{container_name}}`. 3. **Look Inside the Container:** Which program inside `{{container_name}}` is using memory? Run `docker exec -it {{container_name}} ps aux` and sort by memory. **Possible Solutions:** * Give the container more memory if it's hitting its limit. * Check if your `{{application_type}}` app has a memory problem. * Make sure your app's memory settings (like Java's heap size) fit within the container's limit. **Verify:** * After making changes, run `docker stats {{container_name}}` again and watch if it still crashes.

ProfessionalBest with claude

When a detailed, systematic, and technically comprehensive diagnostic workflow is required, matching the main prompt's depth and technical rigor.

prompt.txt

Role: You are a seasoned Site Reliability Engineer (SRE) specializing in container orchestration and performance diagnostics. Context: We are observing persistent, yet sporadic, `OOMKilled` events for a Docker container, `{{container_name}}`, hosting a `{{application_type}}` service. This pattern suggests potential memory oversubscription, a memory leak, or misconfigured runtime parameters. A structured approach is needed to isolate the root cause and implement a lasting resolution. Task: Construct an exhaustive debugging blueprint for `OOMKilled` container incidents. This blueprint must encompass a symptom description, a prioritized hypothesis list, actionable diagnostic checks with specific shell commands, comprehensive remediation strategies, and clear verification protocols. Address cgroup enforcement, process-level memory profiling, and application-specific memory management. Constraints: * Include precise command-line utilities for both host and container-level memory analysis. * Provide explicit instructions for interpreting diagnostic outputs. * Offer solutions covering Docker/Kubernetes configuration, host resource management, and application code/runtime tuning. * The playbook must guide an experienced engineer through systematic troubleshooting. Output: ### OOMKilled Container Diagnostic Playbook **Symptom:** * `{{container_name}}` experiences non-deterministic `OOMKilled` terminations. **Hypotheses:** 1. **Container Memory Cap:** Cgroup limits are too low. 2. **Application Leak/Spike:** `{{application_type}}` has a memory leak or sudden usage burst. 3. **Runtime Heap Config:** JVM/CLR heap settings are incompatible with container limits. **Checks (with Commands):** 1. **Log Review:** `docker logs {{container_name}}` (for `OOMKilled` indicators). 2. **Cgroup Limits:** `docker inspect {{container_name}} | grep -i 'Memory'` (verify allocated vs. used). 3. **Process Memory:** `docker exec -it {{container_name}} ps aux --sort -rss` (identify top memory consumers). **Remediation:** 1. **Increase Limits:** Adjust Docker or Kubernetes memory allocations for `{{container_name}}`. 2. **App Optimization:** Profile and fix memory leaks in `{{application_type}}`, or tune runtime heap parameters. **Verification:** * Monitor `docker stats {{container_name}}` post-fix; confirm no further OOM events under load.

Short VersionWorks with any model

For quick reference or when an engineer needs a concise, one-paragraph summary of the debugging process without extensive detail.

prompt.txt

To debug an `OOMKilled` Docker container, `{{container_name}}`, first inspect `docker logs {{container_name}}` for the OOM event. Then, check the container's configured memory limits versus actual usage with `docker inspect {{container_name}}` and `docker stats {{container_name}}`. If limits are tight, increase them. Next, look inside `{{container_name}}` using `docker exec -it {{container_name}} ps aux --sort -rss` to identify which process, possibly your `{{application_type}}` application, is consuming the most memory. Address application-level memory leaks or adjust runtime heap settings like JVM's `-Xmx` to fit within allocated resources. Verify by monitoring memory post-fix.

EnterpriseBest with chatgpt

In organizations with strict compliance, audit, or incident management protocols, where detailed reporting and stakeholder communication are critical.

prompt.txt

Role: You are a Lead Incident Commander and Technical Lead for critical production systems. Context: A business-critical `{{application_type}}` service, containerized as `{{container_name}}`, is experiencing recurrent, unpredictable `OOMKilled` terminations. This directly impacts service availability and requires a high-priority incident response and a structured root cause analysis (RCA). Stakeholder communication and audit trail documentation are paramount. Task: Formulate a comprehensive OOMKilled incident response and debugging framework. This framework must detail diagnostic steps, remediation actions, and include considerations for risk assessment, impact analysis, and formal documentation for post-incident review and compliance. Constraints: * Emphasize clear documentation requirements for each step. * Include guidance on assessing business impact and communicating status to non-technical stakeholders. * Address compliance implications of resource stability. * Assume an `{{application_type}}` application, container `{{container_name}}`. Output: ### Enterprise OOMKilled Incident Response Playbook **Incident Trigger & Initial Assessment:** * `{{container_name}}` OOMKilled; confirm business impact. **Diagnostic Phase (Document Findings):** 1. **Log Aggregation:** Review central logs for `OOMKilled` events, preceding errors. 2. **Resource Allocation Review:** Verify `docker inspect {{container_name}}` memory limits vs. observed `docker stats {{container_name}}` peaks. 3. **Internal Process Profiling:** `docker exec -it {{container_name}} ps aux --sort -rss` to identify specific memory hogs within the `{{application_type}}` runtime. **Remediation & Risk Mitigation:** 1. **Staged Resource Increment:** Incrementally adjust container memory limits, documenting changes and rationale. 2. **Application Code/Config Review:** Engage `{{application_type}}` development team for potential memory leak analysis or JVM/runtime heap tuning. **Verification & Closeout:** * Sustained monitoring via `docker stats {{container_name}}` and external APM tools. * Formal incident report completion, including RCA, lessons learned, and preventive measures. Communicate resolution to all stakeholders.

What you'll get

Expected output

Debug Playbook: Container OOMKilled

Symptom:

Container my-webapp is being terminated with OOMKilled status in Docker logs or Kubernetes events.
Terminations are intermittent and not consistently tied to specific load patterns.

Hypothesis List:

Cgroup Memory Limit Exceeded: The container's allocated memory (cgroup limit) is insufficient for its workload.
Application Memory Leak/Spike: The application itself has a memory leak or a transient memory spike that exceeds its available resources.
JVM/Runtime Heap Misconfiguration: For applications using runtimes like JVM, the heap settings are not optimized for the container's memory limit.
Sidecar/Helper Process Memory: Other processes within the container (e.g., agents, sidecars) are consuming unexpected memory.
Host-Level OOM Killer: The host system itself is under memory pressure, leading to the host's OOM killer terminating processes, including Docker daemon or containers.

Checks (with Commands):

1. Review Container Logs for OOMKilled Event Details: * docker logs my-webapp * kubectl describe pod my-webapp-xyz12 -n default (if Kubernetes) * *Interpretation:* Look for OOMKilled messages, exit codes (e.g., 137), and any preceding memory-related warnings from the application. For instance, a Java application might log java.lang.OutOfMemoryError before the container is killed.

2. Check Docker Cgroup Memory Limits: * docker inspect my-webapp | grep -i 'Memory' * cat /sys/fs/cgroup/memory/docker/<container_id>/memory.limit_in_bytes (replace <container_id> with actual ID from docker ps for my-webapp) * *Interpretation:* Compare Memory (limit) with MemoryUsage (current usage). If usage is consistently near the limit (e.g., 90% or more) just before an OOM event, the limit is likely too low.

3. Monitor Container Memory Usage Over Time: * docker stats my-webapp --no-stream (for snapshot) * docker stats my-webapp (for real-time monitoring) * kubectl top pod my-webapp-xyz12 -n default --containers (if Kubernetes) * *Interpretation:* Observe memory trends. A gradual, continuous increase suggests a leak. A sudden, sharp spike indicates a transient high-memory operation. Note the peak usage recorded before the OOM.

4. Inspect Process-Level Memory Inside the Container: * docker exec -it my-webapp bash (or sh) * Once inside: ps aux --sort -rss * Once inside: top (then press M for memory sort) * *Interpretation:* Identify which specific process within the container is consuming the most memory. For a Java application, this will likely be the java process. This helps differentiate between the main application and any sidecars or helper scripts.

5. Analyze Application-Specific Memory (e.g., JVM Heap): * If Java is the application type, enable GC logging: -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps * Use jmap or jstat for heap analysis if possible (requires JDK inside container or remote JMX access). For example, jmap -heap <pid> inside the container. * *Interpretation:* Review gc.log for frequent full GC cycles, long pause times, and java.lang.OutOfMemoryError: Java heap space messages. Compare the JVM's configured maximum heap (-Xmx) to the container's cgroup memory limit, ensuring a buffer for non-heap memory.

6. Check Host System Memory: * free -h * dmesg | grep -i oom * *Interpretation:* Check free -h for overall host memory usage. dmesg output can reveal if the host's own OOM killer was triggered, which might indicate a broader system-level memory issue affecting multiple containers or the Docker daemon itself.

Likely Fixes:

Increase Container Memory Limit: If cgroup limits are too restrictive, increase memory and memory-swap in Docker run commands or Kubernetes resource limits (limits.memory). Start with a conservative increase (e.g., 25%) and monitor performance.
Optimize Application Memory Usage:

* Address identified memory leaks in the Java code through profiling tools. * Optimize data structures or algorithms to reduce memory footprint. * For Java, tune JVM heap (-Xmx, -Xms) and garbage collector settings. Ensure -Xmx is significantly less than the container's cgroup memory limit (e.g., 75-85% of the limit) to leave room for non-heap memory, native libraries, and OS overhead.

Adjust Sidecar/Helper Process Resource Allocation: If other processes (e.g., log shippers, monitoring agents) are consuming significant memory, reconfigure their resource usage or move them to separate, dedicated containers.
Scale Out/Distribute Load: If the workload genuinely requires more memory than a single container can efficiently handle, consider horizontal scaling of the my-webapp service or implementing sharding strategies.
Upgrade Host Resources: If host-level OOM is occurring, the underlying server may need more physical RAM or a reduction in the number of containers deployed on it.

Verification Steps:

Deploy Fix and Monitor: Apply the chosen fix (e.g., updated Docker run command, new Kubernetes deployment) and redeploy the my-webapp container.
Observe Memory Metrics: Continuously monitor docker stats my-webapp or kubectl top pod my-webapp-xyz12 for the my-webapp container over several hours or days.
Check Logs for OOMKilled Events: Verify that no new OOMKilled events occur for my-webapp over a sustained period under typical load conditions.
Load Testing: Conduct load tests that simulate peak conditions to ensure the fix holds and prevents OOM events under stress.

Under the hood

Why this prompt works

This prompt generates an effective debugging playbook by employing several prompt engineering techniques. First, role priming establishes the persona of a senior DevOps engineer, ensuring the output reflects an expert-level understanding of containerized systems and diagnostic procedures. This guides the model to produce authoritative and practical advice rather than generic information.

Second, the detailed contextualization of the OOMKilled problem, including its intermittent nature and suspected causes, focuses the model on generating relevant hypotheses and checks. The explicit requirement for specific sections—Symptom, Hypothesis List, Checks, Likely Fixes, and Verification Steps—enforces structured output, which is crucial for a usable playbook. This predefined structure ensures a comprehensive and organized response, preventing the model from simply listing unrelated tips.

Furthermore, explicit constraints such as providing specific Linux commands, interpreting outputs, and addressing both infrastructure and application-level concerns, compel the model to generate actionable steps. The use of placeholders like {{application_type}} and {{container_name}} allows for dynamic, context-specific customization, making the generated playbook directly applicable to a user's scenario. This detailed, constrained, and structured approach yields a systematic diagnostic tool, far more effective than a simple, open-ended query about debugging memory issues.

Model fit

Best AI models for this prompt

Claude

Claude models excel at generating structured, detailed playbooks due to their strong instruction following and ability to synthesize complex diagnostic information into actionable steps. They perform well when asked to provide specific commands and interpretations, making them suitable for technical content like this. Limitations can include occasionally generating commands that need minor syntax adjustments for specific environments. See the full Claude hub for deeper guidance.

ChatGPT

ChatGPT models are effective for producing comprehensive technical guides and debug flows. Their ability to generate clear, step-by-step instructions and elaborate on potential causes and fixes makes them a good fit for this task. They can sometimes be verbose, requiring editing to ensure conciseness, but generally provide solid diagnostic paths. See the full ChatGPT hub for deeper guidance.

Gemini

Gemini models are capable of generating well-organized technical content, including diagnostic procedures. They are particularly strong at integrating different types of information, such as commands, interpretations, and remediation strategies, into a coherent output. Ensure clear separation of sections in the prompt to guide Gemini's output structure effectively. See the full Gemini hub for deeper guidance.

When to use

When OOMKilled terminations are intermittent, making direct cause difficult to isolate.
To systematically differentiate between cgroup memory limits and application-level memory issues.
When fine-tuning JVM heap settings or other runtime memory configurations within a container.
For diagnosing whether memory consumption is a gradual leak or a sudden, transient spike.
If unsure whether the memory pressure originates from the container itself or the host system.

When not to use

When the container is failing due to CPU starvation or other non-memory resource constraints.
If the issue is a clear application crash unrelated to memory (e.g., segfault, unhandled exception).
For network connectivity problems that prevent the container from functioning.
If the container exits cleanly with a non-OOM exit code, indicating a different termination cause.
When the problem is clearly identified as disk space exhaustion, not RAM.

Get more from it

Pro tips

1
Account for non-heap memory when setting JVM `Xmx` values; container limits must exceed `Xmx` to prevent OOM due to native memory usage.
2
Use `docker exec` to inspect process-level memory consumption directly inside the container; `docker stats` only provides aggregate metrics.
3
Collect historical `docker stats` data over several hours or days to accurately distinguish between a gradual memory leak and a sudden memory spike.
4
Always verify the host system's memory status using `free -h` and `dmesg | grep -i oom`; the host's OOM killer can terminate containers.
5
When applying fixes, start with small, iterative memory limit increases. Large jumps can mask the actual root cause and lead to over-provisioning.
6
Use application-specific profiling tools (e.g., `jvisualvm` for Java, `go tool pprof` for Go) for deeper memory consumption analysis within the application.

Don't ship this

Common mistakes

✗ Setting JVM `-Xmx` exactly equal to the Docker or Kubernetes container memory limit.
Fix — Allocate 20-30% headroom for non-heap memory (e.g., native libraries, thread stacks, GC overhead) in addition to the JVM heap.
✗ Only relying on `docker stats` or `kubectl top` for memory analysis without looking inside the container.
Fix — Execute `ps aux --sort -rss` or `top` inside the container to identify specific processes and their individual memory usage.
✗ Immediately increasing container memory limits without first diagnosing the root cause.
Fix — Follow the systematic playbook steps to pinpoint the problem before adjusting resources; this prevents masking issues or over-provisioning.
✗ Ignoring `dmesg` output on the host machine when troubleshooting container OOMKilled events.
Fix — Always check host logs for `OOM` events, as the host's OOM killer can terminate container processes independent of cgroup limits.
✗ Overlooking the memory footprint of sidecar processes or agents running within the same container.
Fix — Use `ps aux` inside the container to identify all memory consumers, not just the main application process.

Frequently asked questions

Q.Does this playbook apply to Kubernetes environments?

Yes, many concepts and checks translate directly. Use kubectl top pod, kubectl describe pod, and examine pod/container logs for OOMKilled events. The underlying cgroup mechanisms are similar.

Q.What if the container is very short-lived after being OOMKilled, making `docker exec` difficult?

For short-lived OOMs, focus on docker logs immediately after termination, dmesg on the host, and any pre-OOM metrics captured by docker stats or monitoring systems.

Q.My application isn't Java; how do I analyze application-specific memory?

Adapt step 5. For Node.js, consider heapdump or built-in profilers. For Python, use memory_profiler. The ps aux command inside the container remains universally useful for process-level insights.

Q.How do I differentiate between a memory leak and a memory spike based on monitoring?

A memory leak typically shows a gradual, continuous increase in memory usage over time. A memory spike is a sudden, often temporary, surge in consumption, frequently tied to specific operations or load patterns.

Q.How much memory should I typically add when increasing container limits as a fix?

Start with conservative increments, typically 10-20% of the current limit. Monitor closely after each adjustment. Avoid large jumps, as they can obscure the actual problem or lead to excessive resource consumption.

Version 1.0Last reviewed July 20, 2026

Reviewed by PromptInFlow Editorial Team