u/13hyperdragoons

I have a self hosted OpenShift cluster on my local machine, Im hoping I can configure remote-write v2 so that I can connect an OTEL Collector with a remote write receiver to read data and transform it to OTEL semantic conventions. This is the documentation I've been reading but unfortunately I found no such documentation how to edit the remote write version. https://docs.redhat.com/en/documentation/red_hat_build_of_opentelemetry/3.9/html-single/configuring_the_collector/index#otel-receivers-prometheus-remote-write-receiver_otel-collector-receivers

This is the config that I use. Im on OpenShift v4.2.0

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring

data:
  config.yaml: |
    prometheusK8s:
      remoteWrite:
        - url: "http://otel-rw.default.svc.cluster.local:19292/api/v1/write"

________________________________
Additional fields which dont work
          messageVersion: V2.0
          headers:
            X-Prometheus-Remote-Write-Version: "2.0.0"

Prometheus Remote write --> OTEL collector -(Metrics in OTEL semantic convention)-> Database

I know that I can just use kubeletstats as a receiver in OpenTelemetry and save myself the hassle of configuring the remote write as well as the remote write receiver, but I just want to know if its possible.

I’m trying to get OpenShift metrics into OTEL semantic conventions while keeping an OTel Collector after Kafka.

My understanding is that if Prometheus Remote Write data is received directly by the OTel Prometheus Remote Write receiver and exported as OTLP, the metrics are converted into OTEL metric format/semantic conventions where applicable.

However, our current pipeline is:

OpenShift Prometheus Remote Write -> Metricbeat -> Kafka -> OTel Kafka Receiver -> OTLP Exporter

The problem is that I don’t think the OTel Kafka receiver can decode Prometheus Remote Write payloads the same way the Prometheus Remote Write receiver does.

Has anyone implemented this architecture successfully with Kafka in the middle?

Specifically:
- Can the Kafka receiver process Prometheus Remote Write payloads correctly?
- Is there a way to preserve/convert to OTEL semantic conventions after Kafka?
- Should the data be converted to OTLP before it reaches Kafka instead?

TL;DR:
How do you convert Prometheus Remote Write metrics coming from Kafka into proper OTEL metrics/semantic conventions using an OTel Collector after Kafka?