r/PrometheusMonitoring

I have built cardamon, a tool that cross-references every metric in your TSDB against Grafana dashboards, alerting/recording rules, and the Prometheus query log, and flags metrics that are scraped but never actually read by anything. Right now it's a standalone binary with a local web UI.

I now want to run it "natively" in Kubernetes either as a Job or as a long running Deployment (as it is right now). Another option is simply to build an official image and run it as an ephemeral container next to Prometheus.

I have a few question to the community regarding a few design decisions:

Would you prefer it as a job running once or as a long running deployment or the ephemeral approach?

Would you even want the generated relabel rules, or is visibility alone enough?

In case you would run it as a job, how should the report be served? As is via a WebUI until the Job times out, as static HTML / JSON without filtering options?

If you store the Prometheus Query Log how do you store it? Imo this can be done either via a ReadWriteMany volume between cardamon and Prometheus, pinning cardamon's pod to Prometheus's node via hostPath + pod affinity or an emptyDir. This would decide where to run cardamon.

We run a large multi-tenant Mimir, and kept hitting the same wall: tenant isolation is all-or-nothing. Tenant B either gets your whole TSDB or nothing. The moment one team needs to see a subset of another team's metrics, your options are duplicating samples into a shared tenant (not viable at volume) or handing over full access and hoping.

So I built authir (Authenticator for Mimir) — an auth gateway that deploys next to an existing mimir-distributed install and adds per-reader grants on top of Mimir's tenancy. Just released 1.0.0 under Apache-2.0.

How it works: the owning tenant grants a slice — metric-name prefixes, required label matchers (e.g. only series labelled user="<reader>"), optionally a label allowlist for responses. authir rewrites every PromQL query before it reaches the query-frontend so it physically cannot select outside the grant, re-parses the rewritten query and compares ASTs as a self-check, and filters responses as a second backstop. Deny by default, no negative rules — blocklists over PromQL are unenforceable, the spec has an appendix on why.

The part I'm happiest with: it works with plain Grafana datasources, no plugin. The token identifies who you are; the Basic-auth username names whose data you're asking for. One tenant, one token, two datasources — "my data" and "what team B shared with me".

Ops details, since that's who this is for: Helm chart that derives everything from your mimir-distributed release name and touches nothing of Mimir's (rollback = uninstall), hot config reload that fails unchanged on invalid config, JSON audit log answering "why did tenant A see this number" (optionally with per-query latency and time range), Prometheus metrics including token-expiry-as-a-timestamp so rotation is an alert instead of a 3am outage. Single static binary in a FROM scratch image, ~6 MB.

Honest limitations: cross-tenant is read-only; queries fan out to one target tenant at a time; metric patterns are prefixes, not regex (deliberately); fuzzing and differential testing against the Go PromQL parser are on the roadmap, round-trip verification is what's in place today. If your idea of fun is smuggling a selector past a query rewriter, the bypass test suite would love your ideas — that's genuinely the review I want.

Repo (code, Helm chart, full design spec): https://github.com/thewillyhuman/authir

r/PrometheusMonitoring

I built a Prometheus unused-metric auditor and need help deciding how it should run in Kubernetes

FlowG: Turning your logs into metrics

authir — open-source cross-tenant read ACLs for Grafana Mimir, enforced by PromQL rewriting