Load Testing an NFT Burn for the Super Bowl: Part 5 — EKS at Scale, IP Exhaustion, Memory Leaks, and Observability
This is Part 5 of the series. Part 1 covers the mission, Part 2 covers load testing strategy, Part 3 covers the infrastructure journey, and Part 4 covers game day results.
Part 3 mentioned IP exhaustion as one of the failure modes we hit scaling the EKS cluster to 300 pods across 60 nodes. The short version there was: VPC subnet too small, fix it and pre-scale instead of reacting. The longer version is worth writing out, because IP exhaustion on EKS at scale is subtle enough to bite you even after you think you’ve handled it — and because it intersects with three other problems we had to solve in the same window: kubelet IP allocation behavior on larger instance types, backend memory leaks that only surface under sustained load, and an observability stack that could actually tell us what was happening in real time.
Why IP exhaustion on EKS is harder than it looks
On most Kubernetes setups, pods share a node’s IP address and the CNI handles the networking multiplexing internally. On EKS with the AWS VPC CNI, that’s not how it works: every pod gets its own VPC-native IP address. That’s a deliberate design choice — it eliminates a NAT hop, lets pods talk directly to other AWS services without address translation, and keeps security group rules simple. The tradeoff is that your cluster’s IP consumption scales with pod count, not node count, and at the pod densities we were running, the math compounds faster than it looks.
How kubelet pre-allocates IPs on larger instances
Here’s the part that catches people off guard: the VPC CNI doesn’t just allocate IPs as pods are scheduled. It pre-warms them. By default, WARM_IP_TARGET and WARM_ENI_TARGET tell the CNI to keep a buffer of unattached IPs ready on each node so pod scheduling doesn’t have to wait for an EC2 API call to complete. That buffer is per node, and on a larger instance type — say an m5.4xlarge or bigger — the buffer is larger, because larger instances can hold more ENIs and more IPs per ENI.
The consequence: the moment a new node joins the cluster, the VPC CNI attaches ENIs and pre-warms IPs well before any pod is scheduled to that node. With 60 nodes, each one claiming a cushion of pre-warmed IPs on top of what their pods actually needed, we were burning through address space fast. Not from pod load, but from the warming behavior itself.
The levers to tune this are environment variables on the aws-node DaemonSet:
env:
- name: WARM_IP_TARGET
value: "2" # keep only 2 unattached IPs per node instead of the default
- name: MINIMUM_IP_TARGET
value: "10" # floor for total IPs on the node (warm + attached)
- name: WARM_ENI_TARGET
value: "1" # keep 1 spare ENI attached per node
Cutting WARM_IP_TARGET from its default (which can be as high as the number of IPs on a full ENI, depending on instance type) down to 2 meaningfully reduced the IP headroom each node claimed on join. For our pre-scaled fleet, that was the difference between the subnet having enough room and running dry as nodes came up.
The right values depend on your pod churn rate. If pods are starting and stopping frequently, a low WARM_IP_TARGET means more frequent ENI/IP attach operations, which adds latency to pod scheduling. For us — pre-scaled, static fleet, pods up and warm before the event — a low warm target was fine. For a cluster doing continuous deployment with fast pod turnover, you’d want a higher value to keep scheduling snappy.
IP prefix delegation: more headroom, different math
IP prefix delegation is the other option. Instead of allocating individual IPs to ENIs, the VPC CNI attaches /28 CIDR blocks (16 IPs each). This increases how many pods a single node can support without requiring a proportionally larger subnet — a useful lever if adding subnet space isn’t straightforward.
Prefix delegation does not eliminate the warming behavior, though. Each node still pre-attaches prefix blocks ahead of pod scheduling. The blocks are bigger, so the headroom is larger, but if your subnet is tight and you have a lot of nodes, you can still exhaust it. We ran with prefix delegation enabled and the WARM_IP_TARGET tuning above as a belt-and-suspenders combination.
Broader EKS best practices we learned at scale
IP allocation was the headline failure, but the three weeks of load testing surfaced a handful of other operational gaps that are worth calling out.
Resource requests and limits are not optional at scale
Before we started tuning, several of the backend pods had either missing or underspecified resource requests. In a low-load environment that’s invisible — the scheduler places pods where there’s room and the node has enough buffer that nothing fights over it. Under sustained high load, it surfaced as unpredictable pod evictions: the node kernel starting to OOM-kill processes because actual memory usage wasn’t reflected in the scheduler’s view of what was claimed.
The fix is mechanical but important: every pod needs explicit CPU and memory requests that reflect actual baseline consumption, and limits that reflect the ceiling under load — not guesses, and not left blank. We ran kubectl top pods and compared it against the requests in the manifest after each load test round to close the gap.
Pod Disruption Budgets keep pre-scaling honest
Pre-scaling to 300 pods ahead of the broadcast was the right call, but it introduced a new risk: a stray kubectl drain or a node replacement during the maintenance window before the event could pull pods out of the fleet faster than PDBs (or their absence) would catch. We added PodDisruptionBudgets with minAvailable set high enough that any voluntary disruption during the pre-scale window would require explicit override rather than silently evicting capacity.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: burn-backend-pdb
spec:
minAvailable: "80%"
selector:
matchLabels:
app: burn-backend
This doesn’t protect against involuntary disruption (node failures), but it does protect against operational mistakes during a high-stakes window.
Karpenter vs. Cluster Autoscaler for burst workloads
We used Cluster Autoscaler during this project, pre-scaling manually rather than relying on it to react. If we were doing this again, Karpenter would be the first thing we’d evaluate as a replacement.
The core difference that matters for this kind of workload: Cluster Autoscaler works at the node group level, which means scaling decisions are constrained by the instance types and sizes you’ve pre-defined in each group. Karpenter provisions nodes directly from EC2, picks the right instance type for the pending pods, and can collapse underutilized nodes more aggressively. For a workload with a hard pre-event deadline and a need to be exact about fleet size and composition, Karpenter’s ability to provision exactly the nodes the pending pods need — rather than scaling up a fixed node group and hoping the instance type is right — is a meaningful operational improvement.
The counterargument for pre-scaled, event-driven workloads: if you’re pre-scaling anyway, the autoscaler mostly stays out of the way and the choice matters less. The autoscaler’s limitations hurt most when you’re counting on it to react.
Memory leaks under sustained load
IP exhaustion and EKS tuning were infrastructure problems. The memory leak was an application problem that only became visible under the load profile we were testing against.
Why load tests expose what unit tests don’t
Unit tests run against a clean process. Memory leaks don’t typically show up in a fresh process that makes a few hundred calls — they show up when a process runs for hours under continuous throughput and memory that should be freed isn’t. The standard profile: memory climbs steadily under load, garbage collection pauses get longer as the heap fills, latency percentiles start drifting up (especially p99 and p999), and eventually the process either hits its limit, gets OOM-killed, or starts degrading badly enough that traffic fails.
We saw exactly this during one of the sustained 30-minute test runs. Pod memory was climbing at a rate that would have put it past its limit before the event window closed.
Finding the leak
The first step was correlating the memory growth with what the pod was actually doing. We used a combination of:
- Go’s
pprofheap profiler (the backend was in Go), exposed as an HTTP endpoint during load testing and captured at regular intervals usinggo tool pprof. Diffing heap snapshots from early in a run against snapshots from later in the same run showed which allocations were growing without being released. - Prometheus metrics on GC pause duration and heap allocation rate, which confirmed the rate-of-growth was tied to request throughput rather than startup behavior — the leak scaled with load.
What we found was a caching layer holding references to objects that should have been eligible for GC after a request completed. The cache had no eviction policy and no TTL — in low-traffic conditions it stayed small enough to not matter, but under sustained load it accumulated without bound.
The fix was adding explicit TTL-based eviction to the cache and ensuring the objects being cached weren’t holding references to larger allocations that kept them alive longer than needed.
What “no memory leaks” means for an event with a hard ceiling
For most backend services, a slow memory leak means you notice it eventually in your dashboards and fix it during normal operations. For a 45-second event window preceded by 300 pods pre-scaled and waiting, a memory leak that manifests after 20 minutes of load is a real risk — especially if your pre-scale window involves keeping the fleet up and warm for a while before the traffic actually arrives.
We ran a 60-minute soak test at sustained load specifically to rule this out: hold request rate at the target load for a full hour, watch heap size, GC duration, and pod restarts. A stable or slightly-growing-then-stable heap profile was the pass condition. A steadily growing one meant we had more work to do.
Observability: Grafana, Thanos, and Loki
None of the above — IP allocation behavior, resource limit gaps, the memory leak — would have been catchable without an observability stack we trusted. This was the other significant investment during the three-week build window.
Why Thanos alongside Prometheus
Standard Prometheus has a data retention window. For load testing across multiple days of runs, we needed metrics that outlasted the retention window on any individual Prometheus instance so we could compare run N against run N-3 and actually see whether the memory growth rate was improving. Thanos handles this by using object storage (S3 in our case) as a long-term backend for Prometheus data, with a sidecar that ships blocks to S3 as they’re compacted and a query layer that federates across instances.
The practical benefit during load testing: every run’s metrics were queryable from the same Grafana dashboard, with the same retention, regardless of when the run happened. We could pull up a time comparison between the first run that surfaced the memory leak and the run after the fix and verify the heap growth curve had actually changed.
Loki for log aggregation
Metrics tell you what is changing; logs tell you why. Loki gave us structured log aggregation across all pods, queryable from the same Grafana instance as the metrics. The integration with Grafana’s Explore view meant that when a latency spike showed up in a Prometheus panel, jumping to the logs from the same time window was a few clicks rather than a separate SSH session.
The operational setup — Promtail as the log shipping agent on each node, scraping pod logs and forwarding to a centralized Loki instance — is low overhead and works well with EKS because Promtail runs as a DaemonSet and picks up logs from the node’s container log path without any application-side changes.
During load testing, the combination gave us a specific workflow:
- Prometheus/Thanos alerts when a metric (p99 latency, pod memory, error rate) crosses a threshold during a run.
- Grafana panel zooms in on the spike window.
- Loki query for the same time window from the relevant pods surfaces the specific requests, error messages, or slow operations that correspond to the spike.
- Fix the root cause, re-run, compare the metric trend from Thanos against the pre-fix run.
The dashboards that mattered most
Not all dashboards are equally useful under pressure. The ones we ended up actually watching during both load tests and the event itself:
- Pod memory over time, per pod and aggregated, with a horizontal line at the memory limit. The single most useful signal for the memory leak work.
- Request rate, error rate, and p50/p95/p99 latency on the same panel, all tied to the same time axis. When error rate goes up, you want to see immediately whether it’s accompanied by a latency change or if it’s a clean error (upstream dependency, etc.).
- ElastiCache hit rate and command latency, because the burn counter in ElastiCache from Part 3 was in the critical path — a cache miss or latency spike there would directly affect claim processing throughput.
- VPC ENI attachment count and available IPs per subnet, which we added after the IP exhaustion incident. We wanted a panel that showed available subnet IPs as a percentage of total, with an alert at a threshold we could actually act on before it hit zero.
- Node count and pod count vs. expected, to detect any unexpected scaling behavior during the pre-scale window before the event.
Alert fatigue is real; threshold tuning is the work
Having dashboards and alerts isn’t the same as having useful dashboards and alerts. The first cut of our alert rules fired constantly during load tests — thresholds that made sense at normal traffic looked like emergencies at 1M req/s, and the volume of noise made it hard to notice the alerts that actually mattered.
The tuning work was iterative: run a test, note which alerts fired, decide whether the thing they fired for was actually a problem or expected behavior at this load level, and adjust the threshold or add a for: duration to require sustained threshold breach before alerting. By the end of the testing cycle, the set of alerts that fired during a test was small enough that each one was worth investigating rather than silencing.
Putting it together for sustained load
The title of this post mentions “sustained load” as a distinct concern from peak load, and it’s worth being explicit about why they’re different problems.
Peak load is about surviving a spike: does the infrastructure have the capacity to absorb the instantaneous rate? Sustained load is about staying healthy over time: does memory stay stable? Does error rate drift? Does the cluster maintain the pod count it should? Do any background processes accumulate state that degrades performance over a run of tens of minutes?
For a 45-second event like the Super Bowl burn, peak load mattered more than sustained load in a narrow sense — we weren’t going to be under full traffic for an hour. But the pre-scaled fleet was up and warm for hours before the spike, and we needed to know it would still be healthy when the moment came. A memory leak that manifested at minute 40 of a 60-minute pre-scale period would have been as damaging as one that manifested under load, because the pods would have been degraded before the event started.
The soak test — sustained load for a full hour, monitoring for all the things described above — was the closure condition on the pre-event validation, not just one more data point in the load test progression. Pass the soak test with clean metrics and the fleet is actually ready. Pass only the peak tests and you’re trusting that nothing accumulates during the wait.
That’s the deeper operational story behind the numbers in Part 4. If you’re running EKS at anything approaching this scale — particularly for an event with a hard start time and no ability to debug live — the short version is: tune WARM_IP_TARGET before your subnets run dry, run a soak test specifically to surface memory leaks, and build the observability stack early enough to actually use it for decisions rather than retrospective debugging.