6 minute read

This is Part 4 of a four-part series. Part 1 covers the mission, Part 2 covers our load testing strategy, and Part 3 covers the infrastructure journey.

Everything in Part 2 and Part 3 was preparation. This post is about what actually happened when the ad aired.

The numbers

We had validated the backend against 1.2 million requests in testing. In production, during the actual broadcast, we peaked at roughly 800,000 — comfortably inside the headroom we’d built. The collection itself was 10,000 free Dragon Egg NFTs total, with roughly 4,000 specifically allocated to the on-screen QR code drop, and that QR-code allocation sold out in about 45 seconds.

sequenceDiagram participant TV as Super Bowl broadcast participant User as Viewers participant CF as Cloudflare LB / 3 ALBs participant BE as Burn backend (300 pods / 60 nodes) participant TW as Twitter participant Site as Corporate domain TV->>User: QR code airs User->>CF: Concurrent scan + claim (~800k peak) CF->>BE: Distributed across 3 ALBs BE-->>User: Burn processed / claim confirmed Note over BE: Allocation sells out (~45s) BE->>User: Redirect once sold out User->>TW: Routed to Twitter Note over TW: In-house metrics polluted by bot traffic TW->>Site: Rerouted to corporate domain for clean signal

45 seconds is not a lot of runway to notice a problem and fix it. That number is the reason the rest of this post reads the way it does — almost everything interesting happened in the minutes immediately after sellout, not during the spike itself.

Limit Break’s own read on the night, posted right after:

“No gas war” is doing a lot of work in that tweet, and it’s the one claim in it I can vouch for directly. A typical high-demand Ethereum mint turns into an open bidding war on gas price, because anyone can submit a mint transaction straight to the chain and the network prioritizes whoever pays the most. Ours never got there, because we weren’t taking raw mint transactions from the public at chain speed — the claim was gated and sequenced through the backend and the ElastiCache counter from Part 3, with the chain settling the burn afterward. The gas war didn’t happen because the bottleneck was never the chain in the first place.

What the internet saw vs. what we saw

If you were watching Crypto Twitter that night, the story that circulated was that the QR code led to confusion — that scanning it, for a lot of viewers, landed them on Gabriel Leydon’s personal Twitter profile instead of an NFT claim, and outlets like Decrypt and Front Office Sports covered it as a stumble — millions spent on a Super Bowl spot that, for much of its audience, ended at a founder’s Twitter feed.

From the backend, that’s a fair description of the symptom, but not quite the cause. The claim allocation genuinely sold out in ~45 seconds under real load we’d tested for. The redirect to Twitter wasn’t the system falling over — it was the fallback destination once the allocation was gone, so that the millions of viewers who scanned after sellout landed somewhere active and on-brand (Gabriel Leydon and Limit Break’s own accounts) instead of an error page or a dead link.

Where it got genuinely hard for us, operationally, was what happened on Twitter next.

The Twitter metrics problem

We started out watching in-house metrics sourced from Twitter to understand how the post-sellout traffic was behaving — engagement, click-through, general health of that fallback path. Within a very short window, it became clear those numbers were being distorted by a large volume of automated/bot traffic hitting the same links at the same moment. A national broadcast plus a trending crypto topic is exactly the kind of event that draws bot activity, and Twitter’s own numbers don’t cleanly separate that from real humans, especially in real time and especially under this kind of load.

With almost no time to spare, we made the call to move the fallback destination from Twitter to our own corporate domain, where we controlled the metrics pipeline directly and could trust what we were seeing. That gave us a clean signal for the rest of the event instead of a number we couldn’t verify.

Lessons

A few things from this project that we’d carry into the next one:

  • Test past your target, not to it. Validating to 1.2M when we expected — and ultimately saw — around 800k gave us margin for the parts of the system that don’t behave linearly under load. If we’d only tested to the expected number, we’d have had no buffer for surprises.
  • Pre-scale for step functions; don’t trust autoscaling to react to one. The Part 3 decision to pre-scale to 300 pods across 60 nodes ahead of the broadcast, rather than reactively scaling into the spike, is the single change that mattered most on the infrastructure side.
  • A load balancer/CDN choice that’s “correct” in general isn’t automatically correct for your traffic shape. NLB and CloudFront are excellent defaults — they just weren’t the right fit for a cold, synchronous, nationwide spike, and the only way we found that out was by testing against something that looked like the real event.
  • Don’t let a third-party platform be your source of truth during a live event. Twitter’s engagement numbers looked fine on the surface and were quietly wrong underneath, because of bot traffic we had no control over. Owning the domain and the metrics pipeline for anything you actually need to trust mid-event is worth the setup cost.
  • The public narrative and the engineering reality aren’t always the same story. The press coverage of viewers landing on a Twitter profile was accurate as far as it went — it just wasn’t the whole picture. It’s worth writing down what actually happened while it’s still fresh, which is a large part of why this series exists.

The people behind the numbers

None of the numbers in this series — 1.2M validated, 800k peak, 45 seconds to sellout — happened because of a clever architecture diagram alone. Every round of load testing in Part 2 surfaced a new way for the backend to fail, and every one of those failures — walking away from third-party vendors that capped at 10k req/s, the load balancer swap, the CDN swap, the IP exhaustion, the ElastiCache counter from Part 3 — got found and fixed by the backend and blockchain teams, specifically, going round after round, in the three weeks they had once the vendor path was ruled out.

That work happened on 12–14 hour shifts, on Zoom, across every team involved, for about three weeks straight leading into game day. This series is a technical retrospective, but it’s worth saying plainly: it doesn’t happen without the people who kept showing up to that call.

That’s the series. If you’re building for a similar event — anything with a hard, externally-fixed start time and a traffic curve that looks like a wall instead of a ramp — the short version is: generate load at a scale you don’t think you’ll need, pre-scale infrastructure instead of trusting reactive autoscaling, and make sure whatever you’re using to judge success in the moment is something you actually control.