State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Mon, 03 Nov 2026 01:14:22 UTC
Finished: Mon, 03 Nov 2026 01:38:05 UTC
Exit code 137 is not arbitrary: in POSIX systems, fatal signals result in an exit code of 128 + n. Signal 9 is SIGKILL ($128 + 9 = 137$). Because SIGKILL cannot be caught, blocked, or handled by the JVM process, shutdown hooks never execute, thread dumps are never generated, and your APM agent ceases transmitting telemetry mid-flight.
Here is the exact diagnostic workflow and configuration fix for production workloads.
The behavior documented below was verified against modern container runtimes and JVM environments:
cgroup v2 enabled (/sys/fs/cgroup/cgroup.controllers)-Xmx Accelerates the CrashWhen engineers first observe an OOM event, the immediate reflex is often:
# WRONG FIX: Allocating 90% of container memory to Java Heap
java -Xms1800m -Xmx1800m -jar app.jar
In a container constrained to 2048Mi memory limit in Kubernetes:
# Pod Spec (Vulnerable)
resources:
limits:
memory: "2Gi"
requests:
memory: "1Gi"
The Linux kernel OOM killer (mm/oom_kill.c) tracks the total resident memory consumption (RSS + swap + page cache) of all processes in the container’s cgroup, not just the Java Heap.
A JVM’s total memory footprint ($M_{total}$) is calculated as:
\[M_{total} = \text{Heap} + \text{Metaspace} + \text{Thread Stacks} + \text{Direct ByteBuffers (Netty/NIO)} + \text{CodeCache} + \text{GC Overhead} + \text{Native C/C++ Libraries}\]When you set -Xmx to 1800MB inside a 2048MB cgroup, you leave only 248MB for:
The moment network traffic spikes, off-heap allocations push total process RSS to 2050MB. The kernel immediately invokes oom_killer and sends SIGKILL (9).
To verify that the host kernel dispatched the kill signal, inspect dmesg or /var/log/messages on the worker node:
dmesg -T | grep -E -i "oom_reaper|killed process"
Expected kernel output:
[Mon Nov 3 01:38:05 2026] Memory cgroup out of memory: Killed process 38421 (java) total-vm:4918204kB, anon-rss:2097152kB, file-rss:1248kB, shmem-rss:0kB, UID:1000 pgtables:8204kB oom_score_adj:988
[Mon Nov 3 01:38:05 2026] oom_reaper: reaped process 38421 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Notice anon-rss:2097152kB ($= 2048\text{ MB}$). The kernel killed the process precisely when anonymous pages hit the cgroup ceiling.
Modern OpenJDK releases (Java 11+) are fully container-aware via -XX:+UseContainerSupport (enabled by default). Instead of hardcoding static -Xmx megabytes in your Dockerfile or startup scripts, leverage percentage-based sizing and restrict off-heap headroom.
# In your Dockerfile or K8s Deployment args:
- ENV JAVA_OPTS="-Xms1800m -Xmx1800m"
+ ENV JAVA_OPTS="-XX:+UseContainerSupport \
+ -XX:MaxRAMPercentage=70.0 \
+ -XX:InitialRAMPercentage=50.0 \
+ -XX:MaxMetaspaceSize=256m \
+ -XX:ReservedCodeCacheSize=128m \
+ -XX:+ExitOnOutOfMemoryError"
-XX:MaxRAMPercentage=70.0: Automatically allocates 70% of the container’s cgroup limit to Heap ($2048\text{ MB} \times 0.70 = 1433\text{ MB}$). This guarantees 30% ($615\text{ MB}$) is reserved for Metaspace, thread stacks, and native buffers.-XX:MaxMetaspaceSize=256m: Prevents runaway classloader memory leaks from consuming the remaining cgroup margin.-XX:+ExitOnOutOfMemoryError: If an application-level heap exhaustion occurs, this forces the JVM to exit immediately with an informative exit code and log line before the Linux kernel kills it silently.Align Kubernetes requests and limits to guarantee consistent Quality of Service (QoS):
apiVersion: apps/v1
kind: Deployment
metadata:
name: billing-service
spec:
template:
spec:
containers:
- name: app
image: ideaspot360/billing-service:v1.4.2
resources:
requests:
memory: "2Gi"
cpu: "1000m"
limits:
memory: "2Gi"
cpu: "2000m"
env:
- name: JAVA_TOOL_OPTIONS
value: >-
-XX:MaxRAMPercentage=70.0
-XX:MaxMetaspaceSize=256m
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/dumps/heapdump.hprof
By providing equal requests.memory and limits.memory, Kubernetes assigns the pod a Guaranteed QoS class, minimizing the probability of node-pressure eviction under tight cluster resources.
Running to Evicted without throwing memory OOM warnings or application-level stack traces:
Status: Failed
Reason: Evicted
Message: The node was low on resource: ephemeral-storage.
Threshold quantity: 5368709120, available: 4194304000.
Container app was using 6291456000, requesting 1073741824.
Or subsequently:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Evicted 2m (x3 over 15m) kubelet The node was low on resource: ephemeral-storage.
Normal Killing 2m kubelet Stopping container billing-worker
Unlike CPU throttling (which merely delays response times) or RAM exhaustion (which triggers OOMKilled exit code 137), ephemeral storage eviction immediately terminates the pod and schedules a reschedule. If multiple pods on the same worker node breach storage thresholds simultaneously, cascading node evictions can disrupt cluster stability.
Here is the exact diagnostic workflow to pinpoint runaway disk consumption and configure bulletproof ephemeral storage boundaries.
The diagnostic rules and remediation configs documented here were verified on:
containerd v1.7.15 (with overlayfs snapshotter)gp3 (mounted on /var/lib/kubelet and /var/lib/containerd)A common reaction to storage eviction is tuning logging frameworks (e.g., Logback or Winston) to rotate logs faster:
# WRONG FIX: Assuming standard container logs are the sole culprit
resources:
requests:
cpu: "500m"
memory: "1Gi"
# Omitting ephemeral-storage limits under the assumption that stdout log rotation handles it
Kubelet calculates a container’s local ephemeral storage consumption ($S_{ephemeral}$) as the aggregate of three distinct layers:
\[S_{ephemeral} = S_{\text{container writable layer (rootfs)}} + S_{\text{stdout/stderr logs}} + S_{\text{emptyDir volumes}}\]Even if your container runtime truncates stdout logs at 10MB, applications that:
/tmpwill rapidly bloat the container’s writable layer (upperdir). Once this layer breaches the kubelet eviction threshold, the node drops the hammer regardless of CPU or memory margins.
Before the pod is killed, identify which directory is ballooning in storage. Execute an interactive exec into a running replica:
kubectl exec -it <pod-name> -n production -- df -h
Then identify disk hogs by depth:
kubectl exec -it <pod-name> -n production -- du -ah / 2>/dev/null | sort -rh | head -n 20
Typical output revealing the leak:
4.8G /tmp/report_cache
1.2G /var/log/application.log
320M /app/node_modules
Notice /tmp/report_cache consumed nearly 5GB of the container’s root file system.
emptyDirTo protect the container filesystem from unconstrained writes, explicitly route high-throughput temporary directories (/tmp, /app/cache) to an isolated emptyDir volume.
For workloads generating ephemeral artifacts, mounting emptyDir backed by medium: Memory guarantees supercharged I/O speeds while preventing disk exhaustion on the host EBS root volume:
apiVersion: apps/v1
kind: Deployment
metadata:
name: billing-worker
namespace: production
spec:
replicas: 3
template:
spec:
containers:
- name: billing-worker
image: ideaspot360/billing-worker:v2.1.0
resources:
requests:
cpu: "1000m"
memory: "2Gi"
ephemeral-storage: "2Gi"
limits:
cpu: "2000m"
memory: "2Gi"
ephemeral-storage: "4Gi"
volumeMounts:
- name: scratch-storage
mountPath: /tmp
volumes:
- name: scratch-storage
emptyDir:
# Memory medium uses tmpfs (consumes container RAM limit instead of host disk)
medium: Memory
sizeLimit: "512Mi"
ephemeral-storage Limits: Guarantees that if disk writes exceed 4Gi, kubelet evicts only this rogue pod before it exhausts the host node’s root filesystem and destabilizes neighboring services.medium: Memory for /tmp: Directs all temporary files into a fast tmpfs RAM slice. If the app forgets to clean up temporary exports, it will trip the 512Mi limit cleanly without touching physical disk sectors.ERROR: deadlock detected
DETAIL: Process 21402 waits for ShareLock on transaction 849201; blocked by process 21408.
Process 21408 waits for ExclusiveLock on tuple (42, 18) of relation "orders" of database 16384; blocked by process 21402.
HINT: See server log for query details.
CONTEXT: while updating tuple (42, 18) in relation "orders"
A common anti-pattern is wrapping queries in exponential backoff retry loops. While retrying mitigates transient collisions, it drastically increases query latency tail percentiles ($P_{99}$) and conceals architectural race conditions under peak traffic.
Here is the exact diagnostic workflow to pinpoint deadlocking queries and restructure locking order.
READ COMMITTED (default)pg-promiseConsider two microservice endpoints updating orders and inventory balances concurrently:
# WRONG FIX: Blind retry loops without lock order synchronization
@retry(exceptions=DeadlockDetected, max_attempts=5, backoff=0.2)
def execute_order_settlement(order_id, user_id, amount):
with db.transaction():
# Service A: Locks user balance first, then updates order status
update_user_balance(user_id, amount)
update_order_status(order_id, 'COMPLETED')
# Concurrently executed by automated invoice reconciliation job:
with db.transaction():
# Service B: Locks order status first, then audits user balance
update_order_status(order_id, 'PROCESSING')
audit_user_balance(user_id)
If Service A locks user_balance while Service B locks orders, both workers block each other. After deadlock_timeout (default 1s), PostgreSQL aborts one transaction. Retrying merely repeats the contention cycle under high throughput, creating database CPU spikes.
pg_locksTo inspect live lock wait queues before the deadlock detector terminates transactions, execute:
SELECT
blocked_locks.pid AS blocked_pid,
blocked_activity.usename AS blocked_user,
blocking_locks.pid AS blocking_pid,
blocking_activity.usename AS blocking_user,
blocked_activity.query AS blocked_statement,
blocking_activity.query AS blocking_statement
FROM pg_catalog.pg_locks blocked_locks
JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid
JOIN pg_catalog.pg_locks blocking_locks
ON blocking_locks.locktype = blocked_locks.locktype
AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database
AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page
AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple
AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid
AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
AND blocking_locks.pid != blocked_locks.pid
JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
WHERE NOT blocked_locks.granted;
Whenever updating multiple rows in batch, always sort primary keys in ascending order prior to issuing SQL statements:
- UPDATE accounts SET balance = balance - 100 WHERE id IN (85, 12, 44);
+ -- Sort IDs beforehand in application memory: [12, 44, 85]
+ UPDATE accounts SET balance = balance - 100 WHERE id = 12;
+ UPDATE accounts SET balance = balance - 100 WHERE id = 44;
+ UPDATE accounts SET balance = balance - 100 WHERE id = 85;
When every worker acquires row locks in identical numerical sequence ($12 \to 44 \to 85$), cyclic deadlocks become mathematically impossible.
For complex workflows spanning multiple tables where row sorting is impractical, use PostgreSQL advisory locks:
-- Acquire transactional advisory lock based on user_id hash
SELECT pg_advisory_xact_lock(hashtext('user_account_' || :userId));
-- Proceed with safe multi-table operations
UPDATE accounts SET balance = balance - :amount WHERE user_id = :userId;
UPDATE orders SET status = 'PAID' WHERE user_id = :userId;
-- Lock automatically releases upon COMMIT or ROLLBACK