> For Terraform (at least a few years ago) a badly reviewed PR could cause catastrophic data loss because resources are deleted without requiring an explicit tombstone.
There have been lifecycle rules in place for as long as I can remember to prevent stuff like this. I'm not sure this is a "problem" unique to terraform.
IIRC, the lifecycle hook only prevents destruction of the resource if it needs to be replaced (e.g. change an immutable field). If you outright delete the resource declaration in code then it’s destroyed. I may be misremembering though
I find this statement to be technically correct, but practically untrue. Having worked in large terraform deployments using TFE, it's very easy for a resource to get deleted by mistake.
Terraform's provider model is fundamentally broken. You cannot spin up a k8s server and then subsequently use the k8s modules to configure the server in the same workspace. You need a different workspace to import the outputs. The net result was we had like 5 workspaces which really should have been one or two.
A seemingly inconsequential change in one of the predecessor workspaces could absolutely wreck the later resources in the latter workspaces.
It's very easy in such a scenario to trigger a delete and replace, and for larger changes, you have to inspect the plan very, very carefully. The other pain point was I found most of my colleagues going "IDK, this is what worked in non-prod" whilst plans were actively destroying and recreating things, as long as the plan looked like it would execute and create whatever little thing they were working on, the downstream consequences didn't matter (I realize this is not a shortcoming of the tool itself).
This sounds like an operational issue and/or a lack of expertise with terraform. I use terraform (self hosted, I guess you’d call it?) and manage not only kubernetes clusters but helm deployments with it just fine and without the issues you are describing. Honestly, this is just my honest feedback, I see things and complaints a lot like this in consulting, where they expect terraform to magically solve their terrible infrastructure and automation decisions. It can’t, but it absolutely provides you the tooling to avoid what I think you are describing.
It’s fair to complain that terraform requires weird areas of expertise that aren’t that intuitive and take a little bit of a learning curve, but it’s not really fair to complain that it should prevent bad practices and inexperience from causing the issues they typically do.
Terraform explicitly recommends in the Kubernetes provider documentation that the the cluster creation itself and everything else related to Kubernetes should live in different states.
> The most reliable way to configure the Kubernetes provider is to ensure that the cluster itself and the Kubernetes provider resources can be managed with separate apply operations. Data-sources can be used to convey values between the two stages as needed.
I agree with you (this is something that OpenTofu is trying to fix), but the way I do k8s provisioning in Terraform is to have one module that brings up a cluster, another to print the cluster's Kubeconfig, then, finally, another to use the Kubeconfig to provision Kubernetes resources. It's not perfect but it gets the job done most of the time.
The Google Cloud Terraform provider includes, on Cloud SQL instances, an argument "deletion_protection" that defaults to true. It will make the provider fail to apply any change that would destroy that instance without first applying a change to set that argument to false.
That's what I expected lifecycle.prevent_destroy to do when I first saw it, but indeed it does not.
I think the previous post is saying a resource removed from a configuration file rather than an invocation explicitly deleting the resource in a command line. Of course if it’s removed from the config file, presumably the lifecycle configuration was as well!
Yeah, that's a legit challenge that it would be great if there was a better built-in solution for (I'm fairly sure you can protect against it with policy as code via Sentinel or OPA, but now you're having to maintain a list of protected resources too).
That said the failure mode is also a bit more than "a badly reviewed PR". It's:
* reviewing and approving a PR that is removing a resource
* approving a run that explicitly states how many resources are going to be destroyed, and lists them
* (or having your runs auto approve)
I've long theorised the actual problem here is that in 99% of cases everything is fine, and so people develop a form of review fatigue and muscle memory for approving things without actually reviewing them critically.
This is not a terraform problem. This is your problem. Theoretically, you should be able to recreate the resource back with only a downtime or some services affected. You should centralize/separate state and have stronger protections for it.
What happens if you forget the lifecycle annotations or put them in the wrong place or you accidentally delete them? Last time I checked it was data loss, but that was a few years ago.
The same as in any other language when what you wrote was not what you intended? Sorry, I’m really confused what your complaint here is or how you’d prefer it to work. If you make a sensitive resource managed by any kind of IAC, of course the IAC can destroy it in a manner that would result in irretrievable data loss. The language has for forever put semantics in place to prevent that, and I’m not sure as a power user I’d want it any other way, I’m explicit with what I want it to do and dont want it making crazy assumptions that I didnt write.
like, what happens if you forget to free a pointer in c? sorry for snark but there are an unbelievably numerous amount of things to complain about in tf, never heard this one.
> what happens if you forget to free a pointer in c?
Assuming you mean 'forget' to free malloc'd space referenced by at least one pointer, that's an easy one .. it's reclaimed by the OS when the process ends.
Whether that's a bad thing or not really depends on context - there are entire suites of interlocked processing pipelines built about the notion of allocating required resources, throughputing data, and terminating on completion - no free()'s
surely my salient point is recognized regardless of semantics, but thanks for the correction. To use another example in another post - what happens if you DROP TABLE in sql?
DROP TABLE is explicit. Inadvertently removing a line from a config file and having Postgres decide to automatically "clean up" that "unneeded table" would be a more apt analogy.
"What happens if I turn a table saw on and start breakdancing on it?"
Of course you're going to hurt yourself. If you didn't put lifecycle blocks on your production resources, you weren't organizationally mature enough to be using Terraform in production. Take an associate Terraform course, this specific topic is covered in it.
I'm not familiar with every lifecycle argument but I don't know of any that prevent resources being destroyed if they are removed from the tf file (what the parent was talking about). prevent_destroy, per docs, only applies as long as the resource is defined.
I think the only way to avoid accidentally destroying a resource is to refer to it somewhere else, like in a depends_on array. At least that would block the plan.
>I don't know of any that prevent resources being destroyed if they are removed from the tf file (what the parent was talking about).
Azure Locks (which you can also manage with Terraform), Open Policy Agent, Sentinel rules, etc. will prevent a destroy even if you remove the definition from your Terraform codebase. Again, if you're not operationally mature enough, the problem isn't the tool, it's you.
"Operationally mature" is code here for "the gun starts out loaded and pointed at your foot". It's fine to point out that that's a suboptimal design for a tool.
>Operationally mature" is code here for "the gun starts out loaded and pointed at your foot"
No, it's code for "don't build a load bearing bridge if you don't understand structural engineering."
> It's fine to point out that that's a suboptimal design for a tool.
This isn't "suboptimal" though. If you delete a stored procedure in your RDBMS and it causes an outage, it's not because SQL/PostgreSQL is suboptimal. Similarly if you accidentally delete files from your file system, it's not because file systems are "suboptimal". It's because you weren't operationally mature enough to have proper testing and backups in place.
Defaults matter. I don't know how else to express that a system which fails safe in the face of fallible humans is strictly better, all else being equal, than the alternative.
It doesn't just "get deleted," by default it prompts and warns the user that there is a deletion planned, and the user must manually confirm the deletion. There is no issue.
There have been lifecycle rules in place for as long as I can remember to prevent stuff like this. I'm not sure this is a "problem" unique to terraform.