r/PrometheusMonitoring

I want to use the OpenTelemetry collector to feed a Prometheus compatible backend which supports PromQL but don't have a Prometheus engine that supports PrometheusRule for alerting and recordingrules.

Is there a solution or a OTel plugin that discovers `PrometheusRule` custom resource like the Prometheus operator and add the Alerting- & RecordingRules feature to my PromQL backend?

I just found vmalert, do you have experience with it and whats your opinion in it?

I use ClickHouse as the backend with cerberus for PromQL support.

Sorry for my bad english ✌🏽

Hi everyone,

I am running a backend API deployed across multiple instances (app-server-01 and app-server-02) behind a round-robin Load Balancer.

When testing locally on a single machine, my Prometheus metrics work perfectly. However, in our clustered environment, my PromQL queries are stuck returning 0, even though traffic has definitely hit the endpoints recently.

Here is a generic sample of my /metrics payload from one of the live backend nodes:

text

# HELP app_http_requests_total Total calls
# TYPE app_http_requests_total histogram
app_http_requests_total_count{category="example-tag",status="200",route="/api/v1/resource",server_id="app-server-01",instance_id="instance-01",env="production"} 2

Use code with caution.

 

My Query:

promql

sum by (category) (
  increase(app_http_requests_total_count[1d])
)

Use code with caution.

 

Why I keep getting 0 (What I suspect is happening):

1. Load Balancer Scraping: Prometheus might be scraping the unified Load Balancer URL directly, causing the counter values to bounce up and down as it hits different nodes.
2. Fresh Deployment resets: The code was recently redeployed, meaning the counter initialized directly at 2 on the first scrape, resulting in a flat line (2 - 2 = 0).

How can I properly configure Prometheus to handle this multi-instance architecture so that increase() and rate() compute correctly without constantly hitting 0? Do I need to bypass the load balancer IPs or target service discovery?

Thanks in advance!

I am running Prometheus on an EC2 instance scraping node exporter from other EC2 servers using static private IPs in prometheus.yml.

Trying to understand what is considered production practice:

• Do teams manually update prometheus.yml or use GitOps / IaC to generate it?
• Is EC2 service discovery commonly used instead of static IPs?
• In AWS setups, do people rely on tags (like monitoring=true) for target discovery, or is that just for automation tools?
• Is node exporter always kept on private subnets with SG restricted only to Prometheus?
• Do production setups ever expose node exporter outside VPC (with protections), or always internal only?
• At what scale does static config become a problem, and what is usually the first upgrade path?

Current setup works, but not sure if this is a “lab pattern” or something actually acceptable in production. Here is the screenshot of the config file that I used. I removed the content marked in yellow because that wasn't working. Seeking your suggestions.

https://preview.redd.it/fwcd4ezpt5ah1.png?width=1111&format=png&auto=webp&s=17b25ad5676a7a32a73956c429184d435dd95680

https://preview.redd.it/mbfhga9xs5ah1.jpg?width=1866&format=pjpg&auto=webp&s=690e79d3d89e3dc3f084ab43b3c55513f5d15619

I newly installed the prometheus stack through Rancher and it has a lot of nice alerts out of the box. I have one that is triggering on a PVC that the pod manages its space by itself and is always high utilization. Based on the rule, it seems I should be able to exclude this PVC by adding a label excluded_from_alerts="true". I've tried adding this to my PVC config under metadata/labels and it doesn't work. What am I doing wrong?

kubelet_volume_stats_available_bytes{job="kubelet",metrics_path="/metrics",namespace=~".*"} / kubelet_volume_stats_capacity_bytes{job="kubelet",metrics_path="/metrics",namespace=~".*"} < 0.03 and kubelet_volume_stats_used_bytes{job="kubelet",metrics_path="/metrics",namespace=~".*"} > 0 unless on (namespace, persistentvolumeclaim) kube_persistentvolumeclaim_access_mode{access_mode="ReadOnlyMany"} == 1 unless on (cluster, namespace, persistentvolumeclaim) kube_persistentvolumeclaim_labels{label_excluded_from_alerts="true"} == 1

I know I can edit the rule and have it exclude the specific PVC, but I'd rather do it the proper way instead of updating the rule for any PVCs I don't want to include in the monitoring.

EDIT:

For anyone else trying to figure this out, ChatGPT had me put this into my helm config and it works. You can add other labels as well (nodes, pods, etc.):

kube-state-metrics:
  metricLabelsAllowlist:
    - persistentvolumeclaims=[*]