Don't want to be that guy, but containerization (gitlab omnibus[0]) might actually help you in this case -- in particular the resource utilization caps that you can get with linux containers. While I can't tell you why Gitlab CE is crashing for you, it is much less likely to take down the entire machine from inside a resource-constrained container, and it will probably force you to make the configuration a bit more reproducable as well (and maybe that will help twiddling).

Would love to read about the actual config issues you're having and how you're solving them one by one -- are you writing about it anywhere?

[0]: https://docs.gitlab.com/omnibus/docker/README.html

That isn't my experience. I installed gitlab on a local debian server and it has been running for 3 months straight now.

How much RAM? Are you installing from Omnibus packages? I ask because I have two servers and have near perfect uptime.

64 GB on a 16 core machine that has nothing else to do.

If any, I think this is maybe too much, every worker just spawns more and more workers. I restricted Postgres from taking all RAM, Sidekiq and Puma to low numbers. But there are more..

I follow the instructions straight from Gitlab.

Consider having databases in a separate instance. 2x32GB is better than one 64GB.

It's specially important at this size to have git in a different machine than the database as both are I/O constrained.

I mean this with no judgement: that is a weird hobby.

Is there a reason you do that instead of fixing gitlab or just using something else? Either of those would seem more satisfying to me.

I doubt that they meant it literally. May have been a reference to the xkcd “my hobby” strips. See https://www.explainxkcd.com/wiki/index.php/Category:My_Hobby

I have somewhat similar experience. I have 16GB of RAM, 12 CPU.

You can reduce the number of unicorn workers in `config/gitlab.rb`.

That was a theory I had: That Unicorn leaks and that Unicorn Killer is not killing any of the workers, since it does so all 160 requests and there are no request happening at the moment.

Then I saw the new installation uses Puma ;)

Hi, GitLab product manager here. I'm sorry that we haven't provided a good experience. We try really hard to make sure GitLab deployments "just work", and clearly it does not for either of you.

Both Puma and Unicorn should be killed after they exceed a certain size to avoid this situation from happening. It's possible either this is not working in some situations/configurations, or there is a leak elsewhere although this is the first time I have heard reports of this.

What configuration is being changed from the defaults? Alternatively if you could open an issue with any additional detail we will try to figure out what is happening and fix it: https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues

what are you seeing in the logs? OOM-killer? Also, newer versions use Puma which lower RAM needs if you are seeing Unicorn OOM Killer

Ah, my bad. I have `12.10.6 (833223f2a7f)`. Time to pull, as it really hogged my RAM. The pattern is that it will periodically consume much more resources.

Thanks for the heads-up.

And Gitea is usable when running on a Raspberry Pi

Disabling automatic repository housekeeping might help. It can use lots of RAM for large repositories like the Linux kernel.

The Linux kernel isn’t that bad. If you really want to kill your instance of Gitlab, try Google Chrome with all its public repos. Write a gitlab ci yaml file and configure it to build/test with every push.

https://chromium.googlesource.com/chromium/

So what's the reason for crashing? What's the configuration that you change? I assume it's not random trial and error.