Strangely enough I feel that we tell ourselves again and again that this problem is not as hard, the previous iteration of programs trying to solve it were just written by incompetent people. It can be made so much easier - the program gets built but in the process the developers realize that it has plenty of edge cases not solved by this easier version and the same complexity gets bolted on later.
The fact is, there is an essential complexity that simply can not ever be reduced. But we should also strive to not get blinded by the n+1 failed, bad attempt because true, novel approaches do turn up from time to time, e.g. nix actually solves the dependency hell problem.
Well… many of those tools are written while building those critical systems and scale, so they may include both the learnings as well as the mistakes. And, frankly, most people aren’t building things that are critical or require scale (no matter what they say to make their work sound interesting).
> many of those tools are written while building those critical systems and scale
Very, very few come out of teams working on the real stuff. Both FAANGs and other companies are not going to release their crucial internal tools and not even discuss them in public.
> may include both the learnings as well as the mistakes
"release early" is not something anybody should do for production critical tools like devops stuff.
> frankly, most people aren’t building things that are critical or require scale
100% agree, but if I build ships they have to fit safety standards even if most people never meet rough seas.
Even worse, people don't even bother to look at other options even if it's just to learn from them.
Compare the new hip thing with a different one and people get defensive.