virtualice is a user on You can follow them or interact with them if you have an account anywhere in the fediverse. If you don't, you can sign up here.
virtualice @CobaltVelvet

Do Fucking Backups
410,757,864,530 DEAD INSTANCES

seriously do your backups. things get broken and deleted very easily. and store them on a different server. at home or on s3 or backblaze or even google drive. many good software: duplicity, borg, ... daily is reasonable but even monthly is better than none. and check them once in a while. if you operate something just have backups. it's important.

· Web · 96 · 120

@CobaltVelvet also don't use scaleway, they've burned so many people I know of. this latest tragedy is just the final proof anybody needs.

@nightpool @CobaltVelvet are you talking about a.w.e? is there another instance that died that i haven't heard of? D:

@eq @CobaltVelvet yeah a.w.e was on scaleway. There were a couple other instances that used to use it back in the day, and i think they've killed more mastodon instances then almost any other single cause (second maybe to docker non-persistent volumes)

@eq @nightpool @wxcafe yes it was a.w.e, but i think it's not the only one recently

@nightpool i think i found the "correct" use of scaleway now: easily reproducible app servers. databases just don't belong there for now, it's cheap unstable ram and bandwidth

@CobaltVelvet i can just back up postgres to s3 right

is there a tool to do this easily

@er1n duplicity can backup to s3, and you can call pg_dump | bzip2 before that to get a clean pg dump that you can safely restore after

@CobaltVelvet I love restic ( and backup to a storage VPS, which is well worth the price (4.50 EUR / month for 1TB) for peace of mind.

@CobaltVelvet @Gargron Spitballing... could encrypted backups of each instance be sent to, say, two other instances? Fediverse + Backups = FedUps!

@qubyte @gargron yeah that can be done with borg, but it would need some good organization and documentation/support effort

@qubyte @Gargron @CobaltVelvet this would be great, but I feel like it won't happen because "privacy" 🙄

@mattbk @gargron @qubyte nah encrypted backups is pretty much already standard

it won't happen because
- it's a lot of work to do cleanly
- "any storage service on the internet" is already a cheaper option
- most instances don't have much storage available to begin with

@CobaltVelvet You did forget to mention automation, or did you leave it out on purpose?

@walialu i thought daily implied some kind of automation

but then if someone want to make backups manually everyday i guess it works too :p

@CobaltVelvet I backup my account data every now and then because I'm probably gonna get banned soon, does that count?

@CobaltVelvet no kidding. Um hourly backups to s3 using the backup gem is cheap

@CobaltVelvet @Gargron how tough would it be to bake (optional) s3 database backups in to Mastodon? Preferably with a sanity check of permissions to avoid backing up to a public bucket.

Also, after you do yr backups:

automate them, so you don't have to remember to kick off the backup manually. Or else you'll remember, like, once every eight months. I speak from experience, here. If you can't automate the backups, at least set a regular reminder and write down your *whole procedure.*


Test your backups. Make sure you can *actually restore* from your actual periodic backups.

Admins, back your stuff up right; the federation is counting on you!