Kubernetes OOMKilled — container exceeded memory limit and was killed

What causes this error

When a container's resident memory (RSS) exceeds the memory limit defined in resources.limits.memory, the Linux cgroup OOM killer terminates the container's main process. Kubernetes then records the exit code as 137 (128 + SIGKILL signal 9) and the reason as OOMKilled. The pod restarts according to its restartPolicy, but if the same workload hits the same limit, it enters CrashLoopBackOff. Common triggers: a memory leak in the application that grows over hours or days; a batch job processing a dataset larger than expected; JVM heap set higher than the container limit; Node.js default heap (1.7GB on 64-bit) exceeding a 512Mi limit; or a sidecar container (like Istio envoy) consuming memory the main container needs.

How to fix it

1. 01

   step 1

   Confirm the OOMKilled event and check exit code

   Verify that the termination reason is OOMKilled (not some other crash). Exit code 137 confirms the kernel's OOM killer sent SIGKILL.

   # Detailed pod status showing last termination reason
   kubectl describe pod <pod-name> | grep -A10 'Last State'
   
   # Look for:
   #   Last State:  Terminated
   #     Reason:    OOMKilled
   #     Exit Code: 137
   
   # Check events for the OOM
   kubectl get events --field-selector involvedObject.name=<pod-name> --sort-by='.lastTimestamp'

   Copy
2. 02

   step 2

   Measure actual peak memory usage

   Before changing limits, measure what the container actually needs. Use kubectl top for current usage and metrics-server or Prometheus for historical peaks.

   # Current memory usage per container in the pod
   kubectl top pod <pod-name> --containers
   
   # If you have Prometheus, query peak memory:
   # container_memory_rss{pod="<pod-name>"}
   # max_over_time(container_memory_rss{pod="<pod-name>"}[24h])
   
   # Check node-level memory pressure
   kubectl describe node <node-name> | grep -A5 'Conditions'

   Copy
3. 03

   step 3

   Set memory limits based on measured usage

   Set the limit to 1.5-2x the observed peak usage to allow for normal variance. Set the request to the average observed usage. Never set limit equal to request for variable workloads — it leaves no headroom.

   # deployment.yaml
   spec:
     containers:
     - name: app
       resources:
         requests:
           memory: "256Mi"   # average observed usage
         limits:
           memory: "512Mi"   # 2x average, covers peaks

   Copy
4. 04

   step 4

   Fix runtime-specific memory issues

   Some runtimes need explicit memory configuration to stay within container limits. JVM, Node.js, and Python all have settings that must match the container limit.

   # JVM — set heap to ~75% of container limit
   # If limit is 512Mi, set -Xmx384m
   java -Xmx384m -Xms256m -XX:+UseContainerSupport -jar app.jar
   
   # Node.js — set max old space to ~75% of container limit
   node --max-old-space-size=384 app.js
   
   # Python — no built-in limit, but check for known leak patterns:
   # - Growing lists/dicts that never get pruned
   # - Unclosed file handles or database connections
   # - Caching without eviction (functools.lru_cache with no maxsize)

   Copy

How to verify the fix

• kubectl describe pod shows no OOMKilled events in the last 24 hours.
• kubectl top pod shows memory usage staying below 80% of the configured limit.
• The pod is no longer entering CrashLoopBackOff due to memory issues.
• JVM/Node.js heap settings are explicitly configured to stay within container limits.