I built a Prometheus unused-metric auditor and need help deciding how it should run in Kubernetes
I have built cardamon, a tool that cross-references every metric in your TSDB against Grafana dashboards, alerting/recording rules, and the Prometheus query log, and flags metrics that are scraped but never actually read by anything. Right now it's a standalone binary with a local web UI.
I now want to run it "natively" in Kubernetes either as a Job or as a long running Deployment (as it is right now). Another option is simply to build an official image and run it as an ephemeral container next to Prometheus.
I have a few question to the community regarding a few design decisions:
Would you prefer it as a job running once or as a long running deployment or the ephemeral approach?
Would you even want the generated relabel rules, or is visibility alone enough?
In case you would run it as a job, how should the report be served? As is via a WebUI until the Job times out, as static HTML / JSON without filtering options?
If you store the Prometheus Query Log how do you store it? Imo this can be done either via a ReadWriteMany volume between cardamon and Prometheus, pinning cardamon's pod to Prometheus's node via hostPath + pod affinity or an emptyDir. This would decide where to run cardamon.