Best Twitter comment (grammar errors and all): "And if your full business relies on one tech partner (no offsite backups) your not doing your tech job right."
This is conflating two different things. One point is valid, the other is not.
- No offsite backups? Agreed. Even for a two person team it is sloppy.
- "Relies on one tech partner?" Strongly disagree. Even large enterprises often have a hard dependency on AWS, Azure, Rackspace, or similar. To suggest that a two person team should have deployment plans for multiple independent cloud vendors is just fantastical thinking with no basis in reality.
Plus if they did over-engineer it by making it cloud agnostic, setting up accounts to sit dormant, cold instances elsewhere, etc, people would just criticize them for that inefficiency/wasting time.
Some may have an availability dependency on those services, but if they don't have a full BC and DR plan ready to go within a few hours of losing those service they're not going to be a big enterprise for long.
Are you talking about a small WordPress site or something?
Very, very few tech companies could simply move everything to a new cloud provider in a few hours. I would even hazard a guess that almost none can.
I have all my infrastructure as code and can break it all down and spin it back up in kubernetes clusters in minutes. But due to the quirks of each cloud provider, there are tons of little fixes that would inevitably need to be made.
Not to mention that many companies have way more data than could even be copied over in a few hours.
The topic is having a single tech partner. If the partner is large enough you can have a high level of redundancy by utilizing their cross-zone/geographically distributed services (e.g. US East, EU West, Asia Pacific, etc).
How you got from "single tech partner" to "have no disaster recovery plan" I don't know.
Maybe. But don't forget this is a small company of just two people. Yes the backups should have been off site but relying on just one digital partner at that scale isn't the worst (they're unlikely to have the money or time to federate out to other services).
Yes, ideally they would have already tested their back-up solution, the back-ups would be offsite and, if something like this happened, they could stand-up on another provider. But that just ignores the reality of them being a super tiny business. Almost no one at that size is going to do that.
I disagree. Many 1 and 2 person shops implement proper off-site backups because they understand that loosing their data is a death sentence for their business. Proper off-site backups are neither expensive nor time consuming to establish these days. There is no good excuse for even the tiniest of companies to not implement them.
Actually, you can implement cheap external backups even in a 1-man company.
But I still support that DO here is a 100% liable toward their client. Now the liability between the said client and his own clients is an other matter.
Indeed. I'd urge everyone here to pay attention how many sites/companies are dead in the water the next time there is a full blown AWS outage and see if they are as quick to heave the same criticisms at those fortune 500 companies that they are at this two man operation.
What happened w/ Digital Ocean is inexcusable, and has potentially dire consequences for two individuals livlihood. In the immediate aftermath of such an event, focusing on the devs percieved lack of disaster preparedness seems petty.
I agree. Who are all these people that pull backups, that maybe GBs or TBs in size for offline storage? How does that even work in practical terms in disaster scenarios like this where resolution times are expected within hours and not days?
It doesn't. I would hazard a guess that there are zero medium to large companies out there right now that could swap to a new cloud provider in a few hours.
On the other hand: Get customers and traction before you build a multi-site, fault-tolerant, self-healing, webscale platform that Google would be proud to have.
I think we needlessly shame one-person operations for focusing on actual customer needs instead of ops busywork and yak shaving.
No one that I've seen so far is saying they should have a system that is "muilti-site, fault-tolerant, self-healing, webscale that Google would be proud to have".
They are saying run a simple backup and keep it literally anywhere else.
I think we needlessly hyperbolize "do a backup once a month and keep it somewhere else" into some sort of NSA operation.
Its backups. Its 2019. Its dead-easy and very affordable.
I agree. Putting all your eggs in one basket these days indicates poor decisions or risk assessment.
> Digital Ocean "Trust and Safety"
Does this phrase give anyone else the heebie-jeebies? They deliberately locked his account without looking at previous metrics over the previous months to conclude this was obviously not a concern, and why?
Things that have crossed my mind:
Is their automatic system was poorly tuned?
Was this deliberately initiated? If so, why?
Ideally partners should be trusted (and trustful), in practice, they aren't
Though trusting DO/AWS/GCP, etc is much more reliable, than, let's say, betting your whole business on somebody's proprietary API (like an FB game, an Linkedin API something, etc)
You're correct. But there are also unicorns that don't follow this rule. It's just that their compute activity would never trigger a false-positive, so everyone (except their ops team) is blissfully unaware of their fragility.
That was the stupidest comment to me. This guy is desperate for help and this fart-sniffer is finger-waving someone he doesn't know online with unnecessary platitudes.
