When you start to scale Prometheus, you quickly bump into the problem of cross-shard visibility. Indeed, Grafana can help, as you can add multiple datastore sources in the same dashboard panel, but this becomes harder to maintain, especially if multiple teams have different needs. Keeping track of which shard has which metric might not be trivial when there aren't clearly defined boundaries - while this might not be a problem when you have a shard per team as each team might only care about their own metrics, issues arise when there are several shards maintained by a single team and exposed as a service to the organization.

Additionally, it is common practice to run two identical Prometheus instances to prevent single points of failure (SPOF) in the alerting path - known as HA (or high-availability) pairs. This complicates...