OHSHIT REPORT: A maxed-out database took meh (and friends) offline for a few hours

On the afternoon of July 8th, meh.com — along with our sister stores morningsave.com, sidedeal.com, hammacher.com, and others — went dark for 2 hours and 52 minutes, from 3:01 PM to 5:53 PM CT. If you came by for a deal during that window and got a sad dog instead, this one’s for you.

Here’s what happened, in mostly-plain English.

All of our stores share a behind-the-scenes service we call catalog-service. It’s the thing that knows every product, price, and deal, and every page you load asks it for that info. catalog-service keeps all of that in a database hosted on MongoDB Atlas (mongodb.com) — a managed database service lots of companies use so they don’t have to babysit their own servers.

First, a myth we’d love to bust. A lot of folks assume a store like ours picks one server, sizes it once, and if things get slow, well, we’re just being cheap. That could not be further from how this actually works. Almost every layer of our setup scales itself up and down automatically, all day long — based both on how busy we are and, the more unusual part, on how busy we expect to be.

The first part is reactive. Our stores and services run on Microsoft Azure, and they automatically add more servers the moment things get busy: when a server’s CPU climbs past a threshold, more copies spin up to share the load, then spin back down once the rush passes. Here are the actual rules for one of our services:

And here’s a look at our catalog servers’ own CPU and memory during the afternoon:

The second part is where our weird little business comes in. Almost everything we do is an event — a daily deal, a big email blast, a Mehrathon, or one of our products getting featured on a TV show. Those throw off sharp, very predictable spikes of traffic, and because they run on a schedule, we know they’re coming. So rather than wait to get slammed and react, we scale up ahead of time — automatically pre-warming extra servers before a scheduled event kicks off, then winding them back down when it’s over. Reacting to a spike is table stakes; seeing it coming and being ready is the fun part.

Our database can resize itself too — when it’s working hard, MongoDB Atlas can automatically bump it up to a beefier machine. We lean on all of this because demand is spiky (a great deal drops, an email goes out, a segment airs on TV) and we would much rather spend more to stay fast than run lean and fall over. Which is what makes what happened next a little ironic.

Our web servers talk to the database over “connections.” Think of connections like phone lines into the database: there’s a fixed number of them, and once every line is busy, new calls get a busy signal. At 3:03 PM CT our database hit 99.9% of its limit — 2,994 of 3,000 lines in use. That’s the line pinned flat against the top here, which is Not Great:

And here’s the irony, given all that autoscaling: this time it worked against us. As our stores and services scaled up to handle the load, every new server opened its own fresh batch of connections to the database — so the harder our whole fleet worked to serve you, the faster we burned through those phone lines.

With every line busy, the database started handing out busy signals, and catalog-service couldn’t get the answers it needed. Its own failed-request rate climbed toward 90% and its response times stretched into the minutes:

The raw count of errors it logged tells the same story:

And since every store leans on catalog-service, every store went down with it. Here’s meh’s own response times ballooning into the minutes and its error rate climbing:

…and meh’s raw error count over the afternoon:

For the record, meh’s own servers were mostly sitting on their hands the whole time — they weren’t the bottleneck, they were just stuck waiting on catalog-service, with a burst at the end as everything came back online:

Right about then, our database provider automatically kicked off a maintenance operation on the cluster — you can see it here, started on its own by “System”:

Normally that’s routine — it restarts the database’s servers one at a time. But because the database was already maxed out and gasping, it didn’t settle down. Instead the servers kept restarting and picking a new leader over and over for nearly two hours, and each restart knocked catalog-service’s connections out all over again. You can watch the database’s processor and memory getting thrown around the whole time — spiking and lurching with every restart:

(We’re honestly still not sure what triggered that maintenance at that exact moment — the timing was rotten, and we’re digging into it.)

To stop the bleeding while we sorted it out, we did the thing you saw: we put the stores into maintenance mode so they’d quit hammering the struggling database. Please enjoy our hard-working dog:

Meanwhile we manually upgraded the database to a bigger size — more of those phone lines, plus extra breathing room — and let it calm down. By 5:05 PM CT it was steady again, and from 5:10 to 5:53 PM we brought the stores back one at a time, watching each to make sure it held. By 5:53 PM everything was back to normal.

We also tracked down a specific thing that had quietly been making our database work harder than it should: an inefficient query behind our “current deals” pages that was occasionally taking six minutes to do a job that should take a few thousandths of a second.

We’re fixing that one so a bad afternoon is less likely to snowball like this again.

Sorry for the busy signal, and thanks for sticking with us while we got everyone back online.