Update: the whole Atlassian suite is compromised, data is lost and can't be restored. We're all under a strict no-VPN-and-don't-come-to-the-office-either policy until next week at best.
Lost data can't be restored.
The former system administrator in me is laughing so hard right now.
A whole week has passed by, in the meantime we all had to bring our company laptops to IT to let them do… uh… stuff. All VPN access credentials have (finally) been revoked, everyone had to file a new access request. Apparently all these requests have to go through the CTO for approval, for each of several hundred employees.
All this was on Monday, today is Thursday and I'm still waiting for my access credentials (and wonder how the hell they're going to send them to me).
But ! There are good news ! A new Bitbucket instance has been created ! On a new domain (because of reasons), using aggregated local repositories from developer machines.
Of course the whole PR history is lost, why would you ask? Oh, and it's only reachable through the VPN, for which we don't have credentials yet.
We're still waiting for : pretty much everything else, starting with the rest of the Atlassian suite that also wasn't backed up.
Insider info : that whole attack was… some crypto-ransomware that happily encrypted everything it could find on the VPN.
Yes, including all production.
In any sane company this would have been solved by restoring from backups. Sane companies have backups. This comany isn't sane.
@jkb le téléphone pardi !
@dashie On y trouve un niveau d'incompétence que je ne pensais pas possible, en effet. On parle d'une boîte qui existe depuis plus de huit ans et emploie plusieurs centaines de personnes hein, c'est pas une micro startup qui va couler dans six mois.
@dashie J ene me fais pas de soucis pour eux vu les résultats financiers, ils vont s'en sortir.
Et d'ici là tout ce merdier ne sera plus mon problème.
@jkb Backup is for dull people! 😂
@pilum No tests, no backups, no audits, no contingency plans, no monitoring, no intrusion detection systems. These are for the unbelievers.
@jkb Ohh dang! This looks unhealthy!
Paid time off for you?
@john Pretty much, yes. I can still write some long overdue documentation, now that I can't write code.
Even that would be very, very hard in my employer's setup if I'm not allowed to use VPN.
Mind if I ask what kind of attack it was? I understand you might not be able to share too much.
@john Even if I was allowed to share such info I wouldn't be able to, information is scarce. All we know is some bad actors gained access to the private part of the infra somehow.
Wow! Haven't they heard of backups?? System roll-backs even? They must not be using a very modern filesystem...
@charlesDelaware All production sites had to be taken offline. A week later most of them are back online but tickets from customer support hint at widespread configuration/database issues.
I don't know the details of Microsoft system administration (we're talking about C# backend and TypeScript frontend, SQL Server databases, IIS as a web server, running on Windows Server) but the past year didn't make me want to try it out. The whole thing is a dumpster fire kept ablaze by incompetence.
@charlesDelaware Here's a fun story about a development environment, in three toots, that really sets the mood for this whole company:
You know something is rotten to the core when it takes weeks to get a development server up and running. It's not a lack of manpower, there's more than a dozen IT staff.
@jkb so your production is like, just a windows share somewhere?
@CobaltVelvet It looks more and more like it. It's a full-Microsoft stack everywhere anyway.
@walruslifestyle Yes, this is a dizzying level of incompetence.
@jkb I feel like my setup at home for storing pictures of my cat is more reliable than what you described as your production system!
@walruslifestyle My home setup is quite abysmal, but then again there's nothing crucial on there. I don't risk bankruptcy or a lawsuit if my MP3 files are lost.
"Worry not. We are your Overlords and will be directly communicating our successful system recovery with your implanted quantum chip (which you agreed to have implanted). Had you read your read your pre-employment documentation, you would be aware of the following. The quantum chip implanted within you is reliable since it relies on quantum entanglement (please consult your hiring documentation). -HR"
...or, it might be easier to mitigate threats through FOSS software! :)
@charlesDelaware I should not be trash-talking the client, but after a year it's obvious to me that they are way too incompetent to even consider any form of change, especially if it's change away from a Microsoft-and-Atlassian walled garden.
The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!