JKB @jkb@octodon.social

Inferences I'm taking from what I can see about the Discord/GCP outage:

1.) Discord's database and storage is effectively single homed. If they had a hot failover it woulda failed over. If they had a cold failover its cost to fail over is probably high or else they would have.

2.) Discord's DB is intolerant of single-region networking issues. Google reports 8% packet loss in one cluster in one DC, and that knocked Discord offline.

3.) Discord may or may not have backups (they better) but the restoration time for a dataset of their size would be days since it's single-homed.

4.) Discord is reliant on Google Cloud. If they had their infrastructure striped across multiple providers, see point 1.

5.) Discord has no response process for if Google Cloud just completely soils the bed, as noted by their updates saying "All we can do is wait for GCP to fix itself"

6.) Discord has no way to gracefully handle a failure between API service nodes and their monolith DB/cache. Granted the nature of the app limits what could be done, but "Wait and hope the message doesn't redline" isn't good and just increases load in a fragile situation.

Synthesis.) A major incident could knock Discord offline for days, or permanently, depending on the reliability of their backups. Absolutely have backup methods for reaching your important contacts.

Last Oglaf is amazing (SFW)

https://www.oglaf.com/brimstone-vinegar/

-"I shall rip it with my BEAR HANDS"
-"Don't you mean bare ha-"
-*bear sounds*

2000s hackers: My dream is to connect all of my devices to the internet.
2020s hackers: My dream is to disconnect all of my devices from the internet.

people are always like “but if you launch donald trump into the sun with a big cannon then mike pence will take over and he’s just as evil” and to that I say: the elegance of my proposal is that the act of firing the cannon also frees up space in the cannon

Reality is now circular. Get used to riding the merry-go-round.

https://freethoughtblogs.com/pharyngula/2019/12/05/the-news-cycle/

asking the real questions
Screenshot_20191204-175645_Driv…

me: ooh what's "foggy" in French

translator: do you want feminine or masculine foggy?

me: *shuts laptop, sets it in fire, starts a bonfire of French dictionaries*