The prometheus scaling problem is a real one. It stems from the "scrape everything" mantra that everyone in prometheus community follows, we should be rather opting-in on which metrics are useful and drop everything by default. We need to go the distributed-tracing way, have smart agents near your metric source when gets everything locally and then decides what metrics are useful and finally ship them off for storage. We are kinda halfway there with OpenTelemetry but the adoption hasn't started yet. Very curious to see how you guys managed it.