OHSHIT REPORT: Fukobukuro 2: The Unluckier Bag

44

When something goes terribly wrong here at Mediocre we draft up internal memos called "OHSHIT REPORTS". We use OHSHIT REPORTS to help share details across the company about what went wrong, coordinate work across teams for how we're going to fix it, and dive deep into root causes.

I thought I'd publicly share an internal OHSHIT REPORT with some details about our recent Fukobukuro deal.

Background

Everybody loves our Fukubukuro deals. The last Fukubukuro deal we had sold out in 1 minute. That's why we spent most of the day on March 31, 2016 preparing our cloud servers for today's Fukobukuro deal.

We quadrupled the amount of servers running meh.com, tripled the amount of servers running our order checkout process, and doubled the amount of servers responsible for signing users into their account.

It worked... sorta

We launched our Fukobukuro 2: The Unluckier Bag for $5 deal right at 12:00:00.000am ET. For the first 550 milliseconds the site performed as expected and let the first 164 customers click the buy button. After 12:00:00.550am ET we noticed an immediate drop to zero pageviews on our checkout page. It took another 3 minutes and 30 seconds before another order was placed. Even after that point orders were barely trickling in... not even quite reaching a 1 order/second pace.

WTF?!

We checked all our metrics. Page response times were normal.

Throughput had increased as expected but CPU and memory usage were holding steady.

Here's a weird thing though. The numbers of CAPTCHAs being attempted increased by tens of thousands. See if you can spot the difference between March 30, 2016 and March 31, 2016 (Central Time) in this Google reCAPTCHA graph:

This appears to have slowed down access to the buy button and checkout page.

We were able to confirm the issue after seeing an increase in Customer Support cases being created. Here's an export of a few random ones right out of our ticketing system:

Those stupid 50 captchas fucked my chance at buying one these bags. Like my 5th time trying. Fuck your site. No more business from me.


You always make these grab bag deals that sell out in 5 minutes flat. I can never get on in time to purchase one. It is literally 15 minutes into the deal and they're sold out. This is outrageous, you've basically just wasted one days worth of deals and it happens EVERY FUCKING TIME. Can you either a.) Not do these shit deals where you only have 400 of an item or b.) FUCKING GET A GOOD STOCKPILE OF THEM. This happens EVERY SINGLE TIME and you fucking know it's gonna happen so adjust accordingly or not do these deals.


Eat shit, assholes. You owe me a good asshole licking. You guys really suck. Woot is SO much better!

After about 8 minutes we started seeing traffic on our checkout page again and we sold out of the remaining inventory.

Action Items

We need to reduce the number of CAPTCHAs being attempted in order to prevent a similar checkout page slowdown when we put the next deal up for sale at 12:00am ET tomorrow.

Super important we get it figured out quickly. We've got a cool thing to sell tomorrow. Here's a sneak peek: