OHSHIT REPORT: Fukobukuro 2: The Unluckier Bag
44When something goes terribly wrong here at Mediocre we draft up internal memos called "OHSHIT REPORTS". We use OHSHIT REPORTS to help share details across the company about what went wrong, coordinate work across teams for how we're going to fix it, and dive deep into root causes.
I thought I'd publicly share an internal OHSHIT REPORT with some details about our recent Fukobukuro deal.
Background
Everybody loves our Fukubukuro deals. The last Fukubukuro deal we had sold out in 1 minute. That's why we spent most of the day on March 31, 2016 preparing our cloud servers for today's Fukobukuro deal.
We quadrupled the amount of servers running meh.com, tripled the amount of servers running our order checkout process, and doubled the amount of servers responsible for signing users into their account.
It worked... sorta
We launched our Fukobukuro 2: The Unluckier Bag for $5 deal right at 12:00:00.000am ET. For the first 550 milliseconds the site performed as expected and let the first 164 customers click the buy button. After 12:00:00.550am ET we noticed an immediate drop to zero pageviews on our checkout page. It took another 3 minutes and 30 seconds before another order was placed. Even after that point orders were barely trickling in... not even quite reaching a 1 order/second pace.
WTF?!
We checked all our metrics. Page response times were normal.
Throughput had increased as expected but CPU and memory usage were holding steady.
Here's a weird thing though. The numbers of CAPTCHAs being attempted increased by tens of thousands. See if you can spot the difference between March 30, 2016 and March 31, 2016 (Central Time) in this Google reCAPTCHA graph:
This appears to have slowed down access to the buy button and checkout page.
We were able to confirm the issue after seeing an increase in Customer Support cases being created. Here's an export of a few random ones right out of our ticketing system:
Those stupid 50 captchas fucked my chance at buying one these bags. Like my 5th time trying. Fuck your site. No more business from me.
You always make these grab bag deals that sell out in 5 minutes flat. I can never get on in time to purchase one. It is literally 15 minutes into the deal and they're sold out. This is outrageous, you've basically just wasted one days worth of deals and it happens EVERY FUCKING TIME. Can you either a.) Not do these shit deals where you only have 400 of an item or b.) FUCKING GET A GOOD STOCKPILE OF THEM. This happens EVERY SINGLE TIME and you fucking know it's gonna happen so adjust accordingly or not do these deals.
Eat shit, assholes. You owe me a good asshole licking. You guys really suck. Woot is SO much better!
After about 8 minutes we started seeing traffic on our checkout page again and we sold out of the remaining inventory.
Action Items
We need to reduce the number of CAPTCHAs being attempted in order to prevent a similar checkout page slowdown when we put the next deal up for sale at 12:00am ET tomorrow.
Super important we get it figured out quickly. We've got a cool thing to sell tomorrow. Here's a sneak peek:
- 54 comments, 73 replies
- Comment
well I hit the buy button 3 seconds after 1200 eastern. Still I got nothing...
@bigtom67 me too, but the chance of me being a robot at that time was way too high, so I continue to prove I was not a robot.
@bigtom67 yeah I hit the buy button exactly at midnight est and all I got was we still think you are a robot garbage
Oh my gosh. Tomorrow's item could be anything...it could even be a boat.
I mean..I really doubt it's a boat...but still...the possibilities!
@studerc you got us, we're going to sell the just-announced Tesla Lakester
@harrison I dont believe that is a real thing. Not going to google..not going to google, screw it I'm googling just in case. Dammit Tesla Lakester doesn't appear to be real.
I was going to say that no OHSHIT report was needed since everything went as planned (although those 164 that made it through without the captcha nightmare should maybe be investigated) but this was a great read. Thanks again, meh!
@stardate820926 I checked myself out... nope, still good. :sarcasm:
Even if it's not truly an "OSHIT", it's still cool to see the analysis of how the event went. Thanks, @shawn.
@stardate820926 Guess I was one of the 164! I thought it went so smoothly I went to my Wife's laptop and instantly hit The Wall of Horror. Tried to complete the captcha course but quite after maybe 10.
So much rage in the emails....
pssh. you people and your inferior 2 prong plugs. why not just completely screw us over and replace all our ketchup with catsup?
@carl669 As it happens, they are fresh out of cats...
@carl669 "I don't want to talk to you no more, you empty-headed animal food trough wiper! I fart in your general direction! Your mother was a hamster and your father smelt of elderberries!"
@cwolfpack3 what do you mean? African or European food trough wiper?
You know that sound, when you hit a golf ball and you just know it's gonna be a great drive? This felt like that sound. Well done, meh and @shawn.
The best part about this OHSHIT report is that I can contribute to it.
The ridiculous amount of reCAPTIA embedded caused my BROWSER to crash on Android, but the wife's iOS based browser worked just fine. Heck, once I borrowed her phone I was even able to get to the order screen...just in time for it to sell out.
Well played Meh. Well played indeed.
@Ellipse yea same here mobile safari on iOS 8.4.1 loaded everything fine
@Ellipse Chrome on my 2013 MBP running Yosemite was smooth sailing. I did watch apple time count down to the bewitching hour and clicked on the bookmark to the Meh homepage at 12:00:02. Not knowing about the debacle that was about to hit other Meh users, I stoopidly clicked the Meh face first, then clicked to buy ... old habits die hard. But I was lucky: only hit a captcha on the final sale page and it did not require clicking little pictures or typing anything. Just a click on the circle and all went through. I also have Windstream as my ISP at 25 mbps. Pretty much middle-of-the-road everything. Nothing fancy. Got my first ever bag from Meh. Been trying for a year.
I had a different experience. Initially when I went to the payment page, it told me to verify that i was not a robot but there was no captcha. I thought that was the joke so i hit the buy it button....a lot...then i thought that was the joke and gave up...for a few seconds. i reloaded the page and the standard captcha showed up. Clicked it and i was able to finalize my purchase. All of this occurred between 9:00 and 9:01.
there were 164 people that didn't have to do the captchas? And I thought I was right on top of that at midnight. According to my Microsoft clock, anyway. Those people that win multiple Fuko's what is their secret? I won't tell anyone.
@wew being lucky and hitting F5 at just the right time.
@wew BOC and Fuku has always been exactly that - sheer dumb luck of the draw. It's hitting the page at the exact right time is all. Browser doesn't matter, OS doesn't matter, mobile or not doesn't matter etc. Browser might have a bit in this unique case due to the captchas but really it's all just pure random luck. Sort of like what it takes to be the 11th caller through to the radio station - just persistence and then random luck to have your call connect at the exact microsecond after another one disconnects.
I even re-synced my clock just a few minutes before -- as if that was going to matter.
In this case, browser did matter. Those of us who got the "copy and paste this code" version of the Captchas were at a decided disadvantage. 
@andrd I'm not so sure. Started hitting my F5 button, continually, a few seconds before midnight, when it was still on the sale item for the 31st and I wasn't able to make a purchase either.
OHSHIT from my house. I got to the page before @humper and had all the CAPTCHAs. He, however, managed to click buy without seeing a single FUCKING BODY OF WATER. Now, who's making me some pancakes?
@Thumperchick I had to find pancakes.
@Thumperchick I had commercial trucks, pancakes, grass, bodies of water, mountains, signs, etc after 15 of them then got error on page. When I first tried to buy the bag it said we still think you are a robot then I had the never ending captchas garbage :(
The complaints are obviously twitter bots, Tay's kids perhaps?
@caffeine_dude not enough racism.
Hotdamn! I will be here at 12:00am buyin' me some ungrounded 'lectricity!
Good job, @shawn.
And I've been looking everywhere for one of those black plugs for my kitchen. Can't wait!
@KDemo Power strips. He's going to be selling white power strips, or surge protectors.
So I do have to say - many months ago I got my gf hooked on Meh. She checks every night at Midnight. Twice since she's been checking a Fuku came up. The first one she had gone to sleep early and missed it. The second one it was sold out before she could buy.
So last night I had a feeling it was coming so we left a concert a few minutes early and were sitting in my car waiting for Midnight to roll over, phones in our respective hands and logged in.
Midnight hits, I saw the page first and yelled 'It's up!' and we furiously start working our way through.
She made it through first and managed to buy her very first Fuku/Fuko. I'm not sure I've ever seen her so excited.
She was bouncing up and down in her seat hi-fiving me and yelling like a lunatic. It was hilariously great to watch her getting so excited about buying what may or may not be crap.
It's also an amazing testament to the wonderful craziness you all have managed to cultivate here. Well done and thank you for a really fun night.
She also got an awesome confirmation number. I forget it now but it ended in sidecar which I just thought was wonderfully random.
@Bingo All this excitement and no video of it?
@Bingo Oh, also, awesome story, thanks for sharing!
@MEHcus Ha. Yeah that would have been great but what the hell do you think I was doing with my own phone! I was busy trying to get my own!
@Bingo
Only thing that would have improved the story - if someone who got one was nekkid or covered in sugar or something.
Happy congrats.
@f00l I'm usually nekkid and covered in caramel when logged in here, if that helps.
*Disclaimer: I ran out of caramel last night.
@Bingo Have her post it on the forum! Get her joining the fun here!
@f00l Who says we weren't?
@Bingo
Thank you very much. ;)
I'd almost rather read an OHSHIT report than win a fuku or fuko. Thanks for sharing the pain.
Fantastic OHSHIT report. I lost to the reCaptcha hell, but it was more entertaining than having meh load straight to a sold out page.
would play again
Yeah I think you guys need fewer Captchas going forward. Makes it difficult for people to buy the item otherwise. Live and learn!
@shawn
Beautiful. I got in before the Captcha-pocalypse, so I found it hilarious. Even moreso because on other sites now it's tripping the alarm, making me click more trees and cats.
hold on a minute @shawn ... if i'm reading this right, the wall of captchas wasn't part of the plan??? Haha if it wasn't, it should be for all future fuku(o)s
Oh and I can't wait to buy that adapter that turns my plug sideways
Computer browser crashed (OK it was Chrome on XP) Android browser flashed continuous captchas as I solved them so no bag for me this time but love the report.
164 people ordered in half a second? The fuck?
But, what if I AM a robot? Bigots...
I'm amazed to be in the 164! Probably should've used that luck on a lottery ticket though...
@anabsa I was also one of the 164... Then i told my friend to log on and his browser crashed XD
@mehhead 164!
I'm glad there were more than a hundred... my odds of getting lynched are a lot lower now.
I fought through all of those stupid Captchas, got to the Buy page, and got a message that the item had sold out while I was processing the page. GRRRRRR!!!
@katjabee Ditto here. I thought the maze of captchas was an April Fool's joke. I got through them and got to the order page, but missed it by two minutes.
@katbyter I thought the same. That getting through the captchas was the April Fool's joke and that after you managed to eliminate all the captchas, you could make a purchase.
@shawn
As you mentioned, you need to reduce the number of CAPTCHAs being attempted in order to prevent a similar checkout page slowdown again.
I think that the solution here is to have some sort of completely automated public Turing test to tell computers and humans apart BEFORE displaying each of the CAPTCHAs tonight only to humans.
Just... Just... I don't think it's fair if you have to pick sandwiches out of a line up when the pictures are grainy, small, out-of-focus, and there is a F@#$!NG GREY SANDWICH. Seriously who eats grey bread? I mean, at the point which it turns grey, one can say that it is no longer a sandwich but now is a poisonous lump of anaphylactic shock.
Two questions:
1) Apart from the 164 unlucky SOBs who had zero CAPTCHAs, how many CAPTCHAs did the other unlucky SOBs who bought a Fukobukuro #2 have to wade through?
2) How many Fukubukuro #2s were there in total? I would check the front page, but I'm done with CAPTCHAs for a good long while.
@a13z Zero for me.
Took 20 minutes to refresh just the images on the bazillion captchas. By then, of course, the good-for-two-minutes captcha wasn't good, and wouldn't refresh. I hate you guys soooo much...
When I go to Meh right now, I get "Internal Server Error". Chrome Version 49.0.2623.110 (64-bit), Macintosh. I'll stop and restart the browser if it's still a problem.
@madamehardy Restarted browser, cleared cache for last hour, all is well. @#$@#4 intermittent bugs.
@madamehardy Your set up is basically the same as mine, but somehow I was one of the 164 and finally got my first Meh bag.
@cwolfpack3 In that case, the system worked exactly the way it should have. Hurray for you! (not being sarcastic)
I think the database system should only allow 1 podcast to 1 address. This bullshit if hubby and wife grabbing one is ridiculous and selfish. Maybe the libtards think it's ok though. Like Benghazi.
@badger65 what the crazy are you babbling about? The fuku grab is the epitome of the Free Market at work. It is supply and demand! Those with the faster resources get the product they want.
What you are asking for is... more regulation? Then you're calling out liberals? Lolwut?
@Thumperchick That's what's wrong with the world today. Two people who live together both getting fukos. And Benghazi, on a slightly less important note.
And where do podcasts come into this equation? So confused.
@djslack fucking hell, did I buy a podcast?
@Ignorant
@Thumperchick
Ok, somebody ask The Donald.
@f00l The purchase of a podcast will be a crime, and the purchaser will face some form of punishment. The provider of the podcast will be prosecuted. That never happened. (source: Donald Drumpf)
@Thumperchick (psst, I think @badger65 is trolling)
@jqubed or has really bad auto correct and tried to say "Fuko bags"
@OldCatLady
Only the hair is constant.
Okay... I'm still confused. Looking at the write-up, I'd have sworn that the excessive captchas were all part of the plan. I mean, the captcha blocked a part of the story that said something about the CEO leaving... for the day.
I contend that the OHSHIT report is a continuation of the April Fools' joke that started with a metric ton of captchas decorating the page to begin with. FWIW, I did complete them ALL, and ended up with a Fukobukuro - I thought that was all part of the gag!
Are you telling me that spackling the entire web page with captchas was some kind of random mistake, and your OHSHIT report's conclusion was "This appears to have slowed down access to the buy button and checkout page."?
Pssst.. Hey, meh - wanna buy some consulting services? I could write up that report for you for just $50....
@Jasongb your second paragraph.
@Jasongb
@Jasongb
(I get so much mileage out of this meme.)
@Jasongb Living with aspergers must be tough
@Thumperchick What's the joke?
@mehrk this entire OHSHIT REPORT is the joke.
I have been buying items here for over a year, including last year's Fuko without a captcha. Finally last night Meh unmasked my deception; I am in fact a robot and Was thwarted before I could deceive them out of yet another item. Well done, Meh - but this isn't the last you've heard from me. I will continue conniving my way past your anti-robot agenda and procuring my share of quality goods at unbelievable prices.
Viva la robot!
@Superjimtendo Kill all humans!
That is a cool thing you're selling today. I don't know if I need it, but it is cool.
@djslack cool is what I drink for breakfast
@mehrk
You never drank me for breakfast, and you ain't gonna start anytime soon.
@f00l dinner perhaps?
After watching the Model 3 reveal on my iPhone (just as I was getting off of a plane) I'm walking around the airport and o shit, its 12 o'clock...lets check meh! I had my headphones on, and a battery stick in my pocket with the wire plugged into my phone. Then I proceed to blindly check off captchas on my iphone for the next 7 minutes while walking around JFK, not looking where I'm going. Low and behold it really ticked me the hell off when I got to the order page and saw "Whoops sorry asshole, we're all sold out"...and you're on the wrong side of the terminal now ;)
...and tonight you give us a bluetooth speaker and a server error. I might've actually been excited if you had put a speaker dock on here.
I'm Kilroy.
@jdeathe
Were you here?
To the first few people who got in without having to do CAPTCHAs... awesome. If I'd been paying attention (instead of watching Outlander), I might have been one of you.
No matter, since the 29 CAPTCHAs were mostly fun. Except for the ones about kayaks: whoever decides on them should really look up the definition of a kayak.
A great job, meh and an excellent prank.
about 30-35 or so for me x 3 browser crashes, hence me NOT getting one. That was a bit excessive; funny on a desktop in hindsight, but 12:00AM EST not so much.....even less so at 12:08AM. Still have mixed feelings about it.
I am ROBOT/DROID.
But, Meh, I ain't your ROBOT/DROID, bitch.
You ain't captcha'd me. Fuck the Fuko!
@f00l "You don't frighten us, Robot droid dogs. Go and boil your bottoms, you sons of a silly person. I blow my nose at you, so-called "female dog of Meh," you and all your silly Robot F-droi-d-ukos!"
I thought the maze of captchas was an April Fool's joke. I got through them and got to the order page, but missed it by two minutes.
So I was laying in bed, checking the site as I do every night a few minutes after 11:00 PM CST on my cheap ass $50 Amazon Fire crap tablet hoping to score some cheap ass meh.com crap and then I get a Captcha. So I fill it out, then see another Captcha. I see the item is sold out but I click the sold out button anyway, more Captchas. Close out the cheap ass Silk browser (it's free, how good can it be?), launch meh.com again. More Captchas. Decide I need to reboot. No luck. Decide to clear the browser cache. No luck. Probably 12:00 AM CST by this time. Consider downloading an alternative browser but who the hell wants to go through that hassle. Could have walked 30 feet to my laptop in the next room but around 10:30 PM I was already pissed that I had to piss, got up, pissed, and I sure as hell wasn't getting back up until I had to piss again in an other hour or so (prostate issues). Finally got pissed once and for all and said, all this effort for what, some mediocre crap from a mediocre company that I would so enthusiastically look forward to receiving in days, but in typical fashion maybe even a week or two later after any excitement turns to disappointment, it finally arrives, I use it and think meh (or @#$%&*+ rip off)? Of course the first website I visited when I arrived at work was meh.com. When I saw the Captchas on my desktop I cursed you jackasses and finished the puzzle, only to vow that I'd never buy any other crap from meh.com until I placed my order for 3 Bluetooth Outlet Speakers of which I'm sure I'll be disappointed with, maybe not even reaching the level of meh but at least taking some satisfaction knowing that though I may have been denied yesterday's offer, I stuck it to you and your partner today confident that my 3x USB speaker score will put you and and your partner out of business for good. Who's laughing now.
@mjhi11
Hope you got to piss.
Why did everyone else have such a fun April Fool's day?
Yeah, agreeing with those customer service complaint excerpts. You need to exponentially increase the amount of fukubukuros you stock.
@Dizavid These weren't fukus. They were fukos. As if that explains everything, eh?
@cwolfpack3
I was fuku fuko'd. Or i was fuko fuku'd.
Only i wasn't here.
How does that work, again? Damn, i forgot again....so confused....
So does your oh shit report offer me a fukobukuro? No? Awww shit.
"we noticed an immediate drop to zero pageviews on our checkout page"
Action Item Cancel extra servers running order checkout.
I love this write up! Part of any good corrective action is what you will do to prevent the problem from recurrence.
If you really wanted to slow people down, you should tell them to run around the house 100 times before they hit the "buy" button.
From a loyal Meh fan who has never been able to get a bag before, thank you @Shawn and co. for this! Finally leveled the playing field against the bots and the multiple bag grabbers so that I could fly through and buy one.
Great write-up @shawn ;)
I was one of the lucky ones to get no captchas (well, except the normal one on the checkout page). I had no idea there was even a joke involved until I started reading the forum and saw the screenshots. I even managed to almost order a second one by accident (and was promptly shutdown by the system telling me I could only order one) because I was shocked that I was able to get one without so much fanfare. Like I said before, there must have been some sort of threshold of lucky people that got to order without the captcha wall (either by timing or by number of fuko purchases prior to them loading the homepage). Whatever the case, it was a funny april fools joke.
To those of you that had to do the captchas, did you have to every one of them on the page before you could order? Or, just enough to uncover the buy it button?
@jsh139 It made you do the poll also. I know you did not need to uncover all the poll responses. Just enough to pick an answer and then submit your poll response. Uncovering the buy button only brought up a message saying they were still unconvinced you were not a robot.
@madmod20061 interesting. I answered the poll, but after the order went through. I read the description and saw the last line that said you had to answer the poll. So I answered it and then tried to order again (in case the first one didn't go through for not following the rules or something). But it said I had already ordered. Cool.
@jsh139 I ended up having to do all the captchas before getting to the order page and then got the sold out message.
@shawn
I'll premise this bit by saying this was the FIRST fuku/fuko event that I actually got to at 12AM EST since every other one that has come up I've missed due to having to go immediately to sleep before/after a 16 hour shift working with a bunch of asshats trying their best to be the "pointy haired boss" to my Dilbert and poison a water supply. Congratulations, you were closer to the "pointy haired boss" in Dilbert than them:
Something tells me this wasn't even close to an issue on April 2nd, more likely people realized you couldn't pass up a Fuko/duke gag on April 1st and feeling (justifiably) ass-raped by the aftermath. Considering the last two days were meh-tastic and the response basically amounting to "we'll make sure this doesn't happen on a day where the same buy rush happening is about as likely as meh following a M.A.P. from a manufacturer" makes you (the site) sound like a bunch of nutless wonders. I get that you probably don't have anything good after one of these events and are just scraping the bottom of the barrel shit you have, but sack up (metaphorically speaking) and just own up and say something like "despite our usual internet awesomeness, we f***ed up about as bad as when FOX cancelled Firefly, we know what happened and this shit won't happen the next time we try something awesome again. Now here's something actually semi-awesome as a deals and as an item. No promises on something new form of fuckery, but we have this one managed."
Do I want something other than y'all growing a proverbial set and offering (actually) awesome deals that we can (actually) buy? Not especially no, won't turn down good stuff, but I usually get that by working smarter/harder than other people and learning from my mistakes/gaps in knowledge. Food for thought.
@seraphimcaduto not sure you get it.
@Ignorant
Ya think?
;)
@f00l and @Ignorant by all means, feel free to enlighten me because clearly I don't. It's past time for the joke to be "funny " to me but there's a large difference between
A.) realizing why I'm a dumbass about this and you getting to (rightfully) rub it in my face about how large of a dumbass I am so that if I ever get "it" I can bask in the glory of your awesomeness of getting it so quickly and
B.) stating someone doesn't get it and being too lame/lazy to at least rub it in my face as to why I (clearly) don't have a refined sense of humor such as yourselves.
Not trying to be mean/rude/lame but at least do the one thing the internet seems to be exceptional at: rub my lack of knowledge in my face.
@seraphimcaduto you see on April the first (April Fools Day to some) people have been known to prank one another. Meh, the clown kings of ecommerce figured they too would get in on this prank business and since every one always complains about the one captcha they need to complete on a Fuku/Fuko purchase, why not make the page nothing but captchas.
They then wrote up a joke OhShit report to make it look like things went wrong, you know because of April Fools Day and all.
tl:dr
It's a funny joke for people that don't think they are entitled to everthing and anything they want.
@Ignorant quite honestly your response (as well as @purplepawprint's) actually did help as to understanding how dumb/literal I was being. I get it now, despite being that guy who laughs at the joke well after everyone else). I don't think I'm entitled to shit personally, just was being EXCEPTIONALLY dense about the gag.
@seraphimcaduto Cool, glad we could shed some light on it for you
Just remember nothing is to be trusted on the Internet on April Fools. Good luck on the next Fuku.
@Ignorant
It's all cool. Thx for updates. This is a place it pays to get the feel of, so hang out w us some more.
@Ignorant
@seraphimcaduto
Sorry, reply above was intended for @seraphimcaduto. No can edit now.
Pretty awesome. I hope you guys do this in the future for all the shit bags. And maybe next year you can have one with no captchas but with the buy button completely disabled. :)
@Al_Coholic
LIKE
I'm totally fine with the captcha gimmick for the Fuku. It was funny and a great use of the Fuku gimmick.
My real issue is that I'm not sure it worked on iOS. I used my iPad, cleared all the Fukus, and then got a message that there were still Fukus remaining. I would have understood a sold-out message, but it was still in the first ten minutes, and I really think there might have been a bug in the code.
I really do appreciate the attempt to do something that doesn't get used up in the first minute up. But walkingaway thinking I might have gotten it if the code worked right doesn't feel superawesome. Looking forward to the next stunt.
@wishlish I wouldn't worry about it being an iOS thing. I got one using iOS this time and so did others.
So this report is itself a joke? I don't know what's real anymore, but I want to know if someone really said they weren't going to give Meh business because of a one-day-a-year prank that will likely not be repeated.
@darksaber99999 I figured that out immediately after I read it. It was pretty obvious.