What observability stack are you using for large scale systems
Hi all!
I am curious what solutions others adopted for observability at scale. We’re having a few EKS clusters with ~8,000 nodes. We are currently using sharded Prometheus with Thanos in each cluster and a common Cortex setup to store all the metrics. We use Splunk for logs and Newrelic for another set of metrics.
I am looking to a common centralized solution for all these items plus traces and data coming from systems like our CI/CD.
I would not want to go with a solution like Datadog that is going to cost a lot.
Does anyone have something built and managed in house? How many people in the team to only do this? What were your drivers to implement it instead of buying?
Thank you,
-JS