The number of employees shouldn't be the deciding factor when you are a tech company that apparently has fortune 500 companies as customers.

I'm not talking 5 9's redundancy. I'm talking grab a backup once a week or something, anything, to help mitigate a scenario like this. According to the thread, they lost ~1 year of data. That should be unfeasible to a company serving customers, let alone Fortune 500 customers.

Disaster recovery planning is key for a technical company to succeed. It is clear they never considered a scenario where there DO account would be closed/compromised/down.

>It is clear they never considered a scenario where there DO account would be closed/compromised/down.

I don't think the chance of DigitalOcean automatically freezing your account to a point where only a co-founder can do something about it has been well publicised.

In all practicality, DO freezing your account has the same effect of DO being down (or closing, etc.), or your account being compromised and you being locked out of it.

A contingency plan should ideally have been in place for a scenario where, regardless of root cause, you have lost access to your DO account.

> In all practicality, DO freezing your account has the same effect of DO being down (or closing, etc.)

Given their size, that is extremely unlikely to happen without warning.

Sure, but them closing combined with the chance of them freezing your account (feasible, considering the topic here) and the chance of account compromise, and the chance they go down for extended maintenance... It is inexcusable not to have a disaster recovery plan for the scenario where you cannot access your DO account.

Imagine you are a customer of this company. Would you be rallying to their defense, "backups aren't needed because the scenarios are unlikely", or would you be angry that the company had zero contingency planning and lost all of your data (or the data you rely upon)?

If you can honestly say, as a (hypothetical) customer of the company in the thread, that you wouldn't care if a company you relied upon has no disaster recovery planning, more power to you. I, however, like to make sure that the companies I'm relying on have some sort of contingency that protects me as a customer.

Plenty of one man companies have Fortune 500 customers. Most are ran like the startup in question, as basic disaster mitigation is overhead. Don't expect the world from one guy keeping a company afloat, unless you enjoy disappointment.

From my experience you don't have fortune 500 companies as a customer as a >50person,>5mil revenue company. You may have employees from such companies paying you via their business credit card, but will never get through real procurement without documented SLA procedures which are required to prevent a scenario exactly like this.

Yeah maybe, but not having an offsite backup is asking for trouble. They just failed to consider that "their site" == "digital ocean's cloud", and that since it's someone else's computers, they could easily be locked out at any time for any dumb reason including this one.

An expensive lesson to learn to take and check backups regularly.

There are certainly a lot of limits on what you can do with such limited resources, but a reasonable backup with a different provider is certainly doable at that small scale.

It won't be entirely up to date when the worst case happens, you'll be unavailable and you'll probably have lost a day of data or so, but you won't have lost everything.

Rclone to AWS or Google or whatever is easy to set up, add a daily dump of your database to the folder you back up. Unless you handle a lot of data, costs are probably not a big factor.

Well if you don't need HA, having a data backup (at least the most important data) outside the service is usually pretty cheap/easy.

Even the database replication is mostly setup and forget these days. Very reliable.

And they already did backups. It's just they did them to the wrong (same service) host.

Their customers are Fortune 500 companies.

Clearly they should be charging more.

It might be that if they do that, they could get undercut by someone else.

True, but it isn’t hard to add S3 as an additional backup destination. Ideally if they had their infrastructure setup as code, it would have made recovery possible.

Agreed but having some backups on a different platform and a fallback plan for deploying somewhere else should be doable.

Are there guides to this that take you through good configs for smaller environments like this? i.e. you have a postgres DB and two web servers. What's the simplest backup process and how do you replicate to another DB VM over on another VPS provider securely?

Stuff like this is relatively simple when you are trying to learn it, but keeping it operation is hard at a small level. Is the less to really use PaaS until it's viable for you to be running a small K8s cluster, or equivalent fleet? Seems really expensive compared to VPSes, but having better guides might help.