Social media share - social media - Media share
socialmedia-share.com

We’re back


Hey folks,

As you may have noticed, the site is back up and running. There are still a few things moving pretty slowly, but for the most part the site functionality should be back to normal.

For those curious, here are some of the nitty-gritty details on what happened:

This morning around 8am PST, the entire site suddenly ground to a halt. Every request was resulting in an error indicating that there was an issue with our memcached infrastructure. We performed some manual diagnostics, and couldn't actually find anything wrong.

With no clues on what was causing the issue, we attempted to manually restart the application layer. The restart worked for a period of time, but then quickly spiraled back down into nothing working. As we continued to dig and troubleshoot, one of our memcached instances spontaneously rebooted. Perplexed, we attempted to fail around the instance and move forward. Shortly thereafter, a second memcached instance spontaneously became unreachable.

Last night, our hosting provider had applied some patches to our instances which were eventually going to require a reboot. They notified us about this, and we had planned a maintenance window to perform the reboots far before the time that was necessary. A postmortem followup seems to indicate that these patches were not at fault, but unfortunately at the time we had no way to quickly confirm this.

With that in mind, we made the decision to restart each of our memcached instances. We couldn't be certain that the instance issues were going to continue, but we felt we couldn't chance memcached instances potentially rebooting throughout the day.

Memcached stores its entire dataset in memory, which makes it extremely fast, but also makes it completely disappear on restart. After restarting the memcached instances, our caches were completely empty. This meant that every single query on the site had to be retrieved from our slower permanent data stores, namely Postgres and Cassandra.

Since the entire site now relied on our slower data stores, it was far from able to handle the capacity of a normal Wednesday morn. This meant we had to turn the site back on very slowly. We first threw everything into read-only mode, as it is considerably easier on the databases. We then turned things on piece by piece, in very small increments. Around 4pm, we finally had all of the pieces turned on. Some things are still moving rather slowly, but it is all there.

We still have a lot of investigation to do on this incident. Several unknown factors remain, such as why memcached failed in the first place, and if the instance reboot and the initial failure were in any way linked.

In the end, the infrastructure is the way we built it, and the responsibility to keep it running rests solely on our shoulders. While stability over the past year has greatly improved, we still have a long way to go. We're very sorry for the downtime, and we are working hard to ensure that it doesn't happen again.

cheers,

alienth

tl;dr

Bad things happened to our cache infrastructure, requiring us to restart it completely and start with an empty cache. The site then had to be turned on very slowly while the caches warmed back up. It sucked, we're very sorry that it happened, and we're working to prevent it from happening again. Oh, and thanks for the bananas.

View Reddit by alienthSource

socialmedia-share

Add comment

Categories

Follow us

Don't be shy, get in touch. We love meeting interesting people and making new friends.

Categories

Social Widget

Collaboratively harness market-driven processes whereas resource-leveling internal or "organic" sources.

Categories

 - 
Afrikaans
 - 
af
Albanian
 - 
sq
Amharic
 - 
am
Arabic
 - 
ar
Armenian
 - 
hy
Azerbaijani
 - 
az
Basque
 - 
eu
Belarusian
 - 
be
Bengali
 - 
bn
Bosnian
 - 
bs
Bulgarian
 - 
bg
Catalan
 - 
ca
Cebuano
 - 
ceb
Chichewa
 - 
ny
Chinese (Simplified)
 - 
zh-CN
Chinese (Traditional)
 - 
zh-TW
Corsican
 - 
co
Croatian
 - 
hr
Czech
 - 
cs
Danish
 - 
da
Dutch
 - 
nl
English
 - 
en
Esperanto
 - 
eo
Estonian
 - 
et
Filipino
 - 
tl
Finnish
 - 
fi
French
 - 
fr
Frisian
 - 
fy
Galician
 - 
gl
Georgian
 - 
ka
German
 - 
de
Greek
 - 
el
Gujarati
 - 
gu
Haitian Creole
 - 
ht
Hausa
 - 
ha
Hawaiian
 - 
haw
Hebrew
 - 
iw
Hindi
 - 
hi
Hmong
 - 
hmn
Hungarian
 - 
hu
Icelandic
 - 
is
Igbo
 - 
ig
Indonesian
 - 
id
Irish
 - 
ga
Italian
 - 
it
Japanese
 - 
ja
Javanese
 - 
jw
Kannada
 - 
kn
Kazakh
 - 
kk
Khmer
 - 
km
Korean
 - 
ko
Kurdish (Kurmanji)
 - 
ku
Kyrgyz
 - 
ky
Lao
 - 
lo
Latin
 - 
la
Latvian
 - 
lv
Lithuanian
 - 
lt
Luxembourgish
 - 
lb
Macedonian
 - 
mk
Malagasy
 - 
mg
Malay
 - 
ms
Malayalam
 - 
ml
Maltese
 - 
mt
Maori
 - 
mi
Marathi
 - 
mr
Mongolian
 - 
mn
Myanmar (Burmese)
 - 
my
Nepali
 - 
ne
Norwegian
 - 
no
Pashto
 - 
ps
Persian
 - 
fa
Polish
 - 
pl
Portuguese
 - 
pt
Punjabi
 - 
pa
Romanian
 - 
ro
Russian
 - 
ru
Samoan
 - 
sm
Scots Gaelic
 - 
gd
Serbian
 - 
sr
Sesotho
 - 
st
Shona
 - 
sn
Sindhi
 - 
sd
Sinhala
 - 
si
Slovak
 - 
sk
Slovenian
 - 
sl
Somali
 - 
so
Spanish
 - 
es
Sundanese
 - 
su
Swahili
 - 
sw
Swedish
 - 
sv
Tajik
 - 
tg
Tamil
 - 
ta
Telugu
 - 
te
Thai
 - 
th
Turkish
 - 
tr
Ukrainian
 - 
uk
Urdu
 - 
ur
Uzbek
 - 
uz
Vietnamese
 - 
vi
Welsh
 - 
cy
Xhosa
 - 
xh
Yiddish
 - 
yi
Yoruba
 - 
yo
Zulu
 - 
zu
en_USEnglish