Login

r/OpenTelemetry

▲ 11 r/OpenTelemetry+1 crossposts

Prometheus Engine for Alerting- & RecordingRules?

I want to use the OpenTelemetry collector to feed a Prometheus compatible backend which supports PromQL but don't have a Prometheus engine that supports PrometheusRule for alerting and recordingrules.

Is there a solution or a OTel plugin that discovers `PrometheusRule` custom resource like the Prometheus operator and add the Alerting- & RecordingRules feature to my PromQL backend?

I just found vmalert, do you have experience with it and whats your opinion in it?

I use ClickHouse as the backend with cerberus for PromQL support.

Sorry for my bad english ✌🏽

reddit.com
u/hell31 — 5 days ago
▲ 11 r/OpenTelemetry

Why aren't the OTel semantic conventions shipped as a versioned, importable package per language?

The API and SDK are clean packages I can pin and upgrade deliberately. But for semconv I still end up copying attribute keys and enum values out of the docs into constants, and re-checking them every time the spec moves, especially the newer gen_ai.* ones. Some languages have a semconv constants package, others lag, and the experimental conventions basically are not importable yet. How are you all handling the drift in practice: generate constants from the YAML yourselves, vendor them, or hardcode and hope the spec does not shift under you? Mostly curious how teams keep app code and the spec in sync without it becoming a chore.

reddit.com
u/dankoverride — 7 days ago
▲ 37 r/OpenTelemetry+10 crossposts

Do you actually need Kafka between your OTel collector and ClickHouse?

Kafka → ClickHouse is the default pattern for OTel pipelines, and for org-wide streaming with replay and many consumers it's a great fit. But for a lot of single-sink observability setups, it's a cluster you're babysitting for no reason.

This post compares where the Kafka layer does real work vs. where you can drop it. It also checks what processing the Collector can or can't do alone (stateful dedup, enrichment-conditional filtering, dynamic sampling, etc.)
https://www.glassflow.dev/blog/opentelemetry-to-clickhouse-do-you-need-kafka?utm_source=reddit&utm_medium=socialmedia&utm_campaign=reddit_organic

Curious what others run:

  1. Kafka buffer,
  2. straight from the collector, or
  3. a lighter processor in between

Leave your comments below, I'd like to discuss the options and understand what folks are using these days!

glassflow.dev
u/Marksfik — 14 days ago
▲ 10 r/OpenTelemetry

We decided to built our own OTLP client for Icinga 2 - honest retrospective and to give you some insights behind the scenes

I'm a dev at Icinga and I recently shipped an OTLP Metrics Writer for Icinga 2. Going in, I had basically zero prior OTel experience. Just want to give you some insights into the last four months to share my experience:

My first instinct was to use the OTel C++ SDK - it's well-established and had everything we needed. But integrating it with our existing codebase turned out to be much harder than expected, and honestly more complex than our use case required. After failing to get it working in a reasonable timeframe, I switched to a tiny OTLP client built on Boost.Beast, which we already used elsewhere in the codebase.

For one, we already used Boost.Beast in our codebase, so it was a no-brainer to use it for the OTLP client as well. Additionally, since the OTel proto spec require proto3 language syntax, we would have had to build the entire OTel SDK from source in order to use our writer with the latest C++ SDK on RHEL 8 and 9 systems, which would not have been feasible for us.

But I didn't see this one coming: proto3 isn't supported by the default protoc on RHEL 8/9, Amazon Linux 2, Debian 11, and Ubuntu 22.04. Two options: ship our own protoc binary, or just disable the writer there. Since most of our customers run RHEL-based systems, disabling wasn't an option - so we ended up packaging our own Protobuf compiler for RHEL 8 and 9. For Amazon Linux 2, Debian 11, and Ubuntu 22.04, the writer is currently unavailable unless you build from source.

In OTel, a service presents itself and its metrics are associated with that service. Icinga doesn't work that way. it's not the one being monitored, it's acting as a proxy for the checkables it monitors. We went back and forth a lot on this one. How do you even represent Nagios-style check results in a way that makes sense in OTel? Shoutout to Markus Opolka (on Github) who provided a lot of useful input on this part.

And just before final reviews, my colleague Alvar Penning (Github) found a severe bug in the OTLP client that caused Icinga 2 to hang on reload. Major refactoring, significant delay. The embarrassing part: the bug was trivial to trigger. If I had reloaded Icinga 2 even once in my dev environment during development, I would have caught it. :P Won't make that mistake again.

__

Four months total (longer than expected), mostly because starting from scratch with OTel means working through a lot of documentation before you can write anything meaningful. Also came out the other end knowing a lot more about Protocol Buffers than I expected.

Happy to answer questions about the metrics mapping or the proto3 packaging approach, or anything else that comes to your mind!

Yonas/ Icinga

u/icinga — 12 days ago