Load Testing an NFT Burn for the Super Bowl: Part 3 — The Infrastructure Journey
This is Part 3 of a four-part series. Part 1 covers the mission, Part 2 covers our load testing strategy, and Part 4 covers game day itself.
Every load test in Part 2 existed to find the next thing that would break. This post is a walk through what actually broke, in the order we found it, and what we changed each time.
The production path settled on three Application Load Balancers (ALBs) in front of the NFT token burn backend. Getting to that steady state took a few wrong turns first.
Detour 1: outsourcing the scaling problem
Before any of the AWS-native detours below, our first approach wasn’t to own the backend’s scaling story ourselves at all — it was to split it into pieces and hand each piece to a third-party vendor who specialized in scaling exactly that piece. On paper that looked like the faster path: let specialists own what they’re best at, and spend our own time on the burn logic itself.
In testing, none of those vendors could sustain even 10,000 requests per second — nowhere close to the order of magnitude we now knew we needed after benchmarking against Coinbase’s precedent in Part 2. That’s not a marginal shortfall; it’s roughly two orders of magnitude short of the ~1M/sec worst case we were planning against, and no amount of incremental tuning on the vendor side was going to close a gap that size in the time we had left.
With about three weeks left before the ad would air, we made the call to walk away from the vendor path entirely and bring the scaling problem in-house — architecture, load balancing, pre-scaling strategy, the counter, all of it, owned end-to-end by our own backend and blockchain teams instead of split across vendors we didn’t have deep enough visibility or control over. Everything from here through Detour 5 is what got built in that window.
Detour 2: NLB, on AWS support’s advice
Our first instinct — reinforced by guidance from AWS support — was to put a Network Load Balancer in front of the backend. NLBs operate at layer 4, handle millions of requests per second with very low latency, and are the textbook answer when someone says “we need to survive a massive, sudden spike.”
On paper that’s the right call. In practice, once we went through AWS’s Infrastructure Event Management process for preparing an account for a major scaling event and looked at how our specific workload behaved under it, the NLB path didn’t scale the way we needed for this traffic shape. The burn flow wasn’t a raw TCP/UDP firehose — it needed the layer 7 features (routing, health checks tied to application state, and the flexibility to shift backends without re-architecting DNS) that an ALB gives you and an NLB doesn’t.
We moved to ALBs, and instead of leaning on the load balancer choice alone to absorb the spike, we put more of the work into optimizing the backend itself to sustain the load for the duration of the event — not just survive an instantaneous burst.
Detour 3: CloudFront, then Cloudflare
In front of the ALBs, we started with Amazon CloudFront as the CDN layer — the natural default in an AWS-native stack. It didn’t scale the way this specific spike needed. Super Bowl traffic isn’t a gradually-warming cache; it’s cold, global, and synchronous, and we needed a CDN/load-balancing layer that could spread that instantaneous load across multiple origins without a warm-up period.
We switched to Cloudflare’s load balancer, distributing traffic across all three ALBs, and combined that with AWS’s special-event scaling support on the origin side. That pairing — Cloudflare absorbing and distributing the edge spike, AWS’s event team helping us pre-provision and monitor the account during the broadcast window — is what actually got us a CDN/LB layer that behaved predictably under a cold, instant, nationwide spike.
(300 pods / 60 nodes)"] end
Testing against Cloudflare without tripping it
Once Cloudflare was in front of the ALBs, load testing it honestly got trickier — a load balancer sitting behind a bot-mitigation layer is, definitionally, going to look a lot like a bot when you point six accounts’ worth of JMeter at it. Two things made that workable:
- We whitelisted Cloudflare’s published IP ranges on the load balancer/proxy side, so traffic legitimately coming through Cloudflare’s edge wasn’t itself the thing getting blocked further downstream.
- We whitelisted the EC2/ECS CIDR ranges for
us-east-1andus-west-2— the two regions our load-testing accounts ran in from Part 2 — on the Cloudflare side, and gave our account manager a heads-up before every large run so a sudden wall of traffic from AWS IP space didn’t get flagged and mitigated as an attack mid-test.
The genuinely pleasant surprise was that none of this required Cloudflare to pre-scale anything on their end. We told them when we were testing as a courtesy, not because their load balancer needed lead time to absorb what we were about to throw at it — it just handled it.
Detour 4: running out of IPs
The next failure wasn’t at the edge at all — it was inside the VPC. As we scaled pods out to meet load-test targets, we started running out of IP addresses. The VPC subnet backing our node group was sized too small, and on Kubernetes with the AWS VPC CNI, every pod gets an IP address from the VPC itself — pods don’t share a node’s IP the way they might on other CNIs. Scale pods out fast enough on an undersized subnet and you exhaust the address space before you exhaust anything else.
This is a well-known sharp edge in EKS: even with IP prefix delegation enabled, a subnet that’s too small will hit a wall, because each node claims IPs in blocks as it scales, and a handful of busy nodes can burn through a subnet’s headroom well before pod count looks alarming on paper.
The fix wasn’t just “make the subnet bigger,” though we did that too. It was a change in strategy:
- Pre-scale instead of react. Rather than trusting horizontal pod/node autoscaling to keep up with a spike that goes from zero to peak in seconds, we pre-scaled the fleet ahead of the broadcast window to 300 pods across 60 nodes — capacity that was already up and warm before the ad aired, not capacity we were racing to create in real time.
- Right-size IP allocation per node. We tuned how many IPs each node was allowed to claim so the pre-scaled fleet actually fit inside the VPC’s address space, instead of the first wave of nodes greedily reserving IP blocks that later nodes then couldn’t get.
That combination — a subnet with enough room, pre-scaled capacity instead of reactive autoscaling, and IP allocation tuned per node — is what let us actually hit the 1.2M-request test scale from Part 2 without the cluster falling over on networking before the application even got a chance to.
Detour 5: the chain was too slow to count against
The last failure we kept hitting at every rate we pushed to wasn’t infrastructure in the traditional sense — it was analytics. We needed to know, in near real time, how many burns had happened so far, both to know when the allocation was gone and to feed live numbers back into the product. The obvious place to get that count is the same place the burn is actually recorded — the blockchain, via our data pipeline — and at every load level we tested, that path was too slow. Block times, confirmation, and pipeline ingestion lag are all fine for an eventually-consistent record of truth; none of them are fast enough to gate a limited allocation that was going to sell out in under a minute.
The fix the backend and blockchain teams landed on was to stop treating the chain as the thing you count against in the moment. We used ElastiCache as a fast, in-memory counter, incremented synchronously as part of the burn request path, completely decoupled from waiting on the data pipeline to catch up with the chain. The chain was still the durable source of truth for the burn itself — that never changed — but the count, the number that decided whether the next request should even be accepted, lived in ElastiCache because it was the only thing fast enough to keep up with the rate we were testing at.
None of the fixes in this post — walking away from the vendor path, the load balancer swap, the Cloudflare migration, the IP allocation strategy, or this one — would have happened without the backend and blockchain engineers who kept finding these failure modes round after round and fixing the specific thing that broke, not just the symptom. That work happened on 12–14 hour shifts, on Zoom, across every team involved, for the roughly three weeks between the decision in Detour 1 to bring everything in-house and game day itself.
Part 4 covers what all of this actually looked like when the ad aired for real.