r/OpenTelemetry

oTel collector Daemonset vs sidecar

I feel like Daemonset collectors have become the de facto standard. Out of curiosity what are some situations in which you opted / would opt for sidecars per deployment?

reddit.com
u/DisastrousBrain5417 — 4 days ago
▲ 32 r/OpenTelemetry+4 crossposts

log4k 2.3.0 — a Kotlin IR compiler plugin that instruments your functions with tracing, logging and metrics

log4k is a coroutine/channel-based logging + tracing + metering library for Kotlin Multiplatform (JVM, Android, iOS, macOS, Linux, Windows, JS, wasmJs, wasmWasi), aligned with the OpenTelemetry model.

The recent addition is log4k-compiler-plugin — a Kotlin IR compiler plugin that rewrites annotated functions at compile time, so the instrumentation boilerplate disappears from your source. It runs on common IR before backend lowering, so the same annotations work on every KMP target — not just the JVM (no AspectJ, no bytecode agent, no reflection).

Setup — one Gradle plugin, no extra config:

plugins {
    id("io.github.smyrgeorge.log4k") version "2.3.0"
}

dependencies {
    implementation("io.github.smyrgeorge:log4k-classic:2.3.0")
}

@Traced — wraps the body in a span (started, ended, marked failed on throw):

@Traced
context(_: TracingContext)
suspend fun loadUser(id: Long): User {
    // ...
}   // span "UserService.loadUser"

The parent span is resolved from what's in scope: a TracingContext param/receiver → nests under its current span; else a TracingEvent.Span in scope → used as parent; else a trace: Tracer member (reused, or synthesized) → new root span.

@Logged — entry/exit/failure logging:

@Logged
fun compute(x: Int): Int = x * x
// → UserService.compute(x = 5)
// ← UserService.compute = 25(12.5 us)

Throwing logs ✗ UserService.compute failed (…) at ERROR with the throwable, then rethrows. If a span is in scope it's attached to every emitted line.

@Timed — call/error counters + a duration histogram:

@Timed(tags = [Tag("tier", "gold")])
suspend fun placeOrder(id: Long): Order {
    // ...
}

Records OrderService.placeOrder.calls, .errors and .duration (ms histogram) — exportable in OpenMetrics line format via SimpleMeteringCollectorAppender.

Details that mattered while building it:

  • suspend and regular functions are both supported; the generated wrapper delegates to inline helpers ( Logger.logged, Meter.Timed.measure, TracingContext.traced), so there's no per-call lambda allocation.
  • The plugin reuses your existing log / meter / trace members if they're thesynthesizes private val _log_ = Logger.of( this::class) under a distinct name,so it never clashes with e.g. an existing SLF4J log.
  • All three annotations work class-level too — annotate the class to instrummember. Per-function annotations override the class defaults, and @NoLog / @NoTime / @NoTrace opt out a single function or the whole class.
  • The metric instrument bundle is created once and cached per name.

The plugin is marked experimental — behavior and API may still change.

Repo: https://github.com/smyrgeorge/log4k

Compiler Plugin: https://github.com/smyrgeorge/log4k#compiler-plugin

Docs: https://smyrgeorge.github.io/log4k/

Feedback welcome, especially on the annotation surface and on cases where thon't pick what you'd expect.

u/smyrgeorge — 12 days ago

Feedback about E2E tests based on OpenTelemetry traces?

Hi everyone,
I have just published my open source project called mtracer and I would like to understand if it’s good idea or what should I change (I’m a new grad).

The idea

Mtracer a CLI tool that relies on OpenTelemetry traces to assert system behavior.

I believe that E2E tests should be:
- Cheaper to write and maintain
- Easier to debug

So this is the workflow:

  1. ⁠You configure mtracer to fetch from your observability backend (currently supporting Jaeger and OpenObserve).
  2. ⁠You define your first .mt.yaml test by specifying:
  3. ⁠Trigger: the first call to the system (for instance, an HTTP request).
  4. ⁠Expected trace and spans: the OTel properties of the trace and spans that you expect your system to generate.
  5. ⁠You run the test and see the results!

What actually happens during the run?

  1. ⁠It parses the mt.yaml file.
  2. ⁠It executes the trigger: mtracer injects a generated traceID into the trigger (for an HTTP request, the traceID is inserted into the traceparent header). Subsequent requests will be correlated to this generated traceID as long as your system has OpenTelemetry set up correctly.
  3. ⁠It fetches the trace matching the generated traceID from the configured observability backend.
  4. ⁠It compares the expected trace with the fetched one.

Many other features are available; check out the documentation to discover all of them: documentation website

I would love to have some feedback from more experienced people than me.

reddit.com
u/Proud-Contact9951 — 13 days ago

A Collector exporter that turns agent traces into a signed, verifiable audit log (now in the registry). Feedback on the approach welcome.

Sharing a component I built and recently got listed in the OpenTelemetry registry: otel-agent-audit.

The idea: as AI agents take real actions, you want a provable record of what happened. Instead of adding a new instrumentation layer, this consumes the gen_ai.* spans your agents already emit and turns them into a tamper-evident audit log, entirely inside the Collector pipeline.

The pipeline:

otlp -> memory_limiter -> agentauditselect (buffers each trace until its root arrives) -> agentaudit exporter (per-trace hash chain -> Ed25519 sign -> seal) -> audit.jsonl + checkpoint.jsonl

A separate verifier CLI checks the whole thing with only the public key, so anyone can independently verify authenticity and integrity without a shared secret.

Things I'd love this community's take on:

- Passive instrumentation as the right model: reusing existing spans rather than asking teams to re-instrument.

- Whether governance/guardrail decisions belong in spans, and how they'd ideally map to semantic conventions. I'm interested in where the GenAI SIG is heading on policy/guardrail signals.

- The single-writer constraint (one Collector instance) that deterministic ordering forces, and whether that trade is acceptable.

Caveats up front: third-party, experimental, not audited. It's observability only, it does not enforce or block. It gives tamper-evidence on honest infra, not protection against an operator holding the signing key.

Repo: https://github.com/surpradhan/otel-agent-audit

It's in the registry under "agent audit" if you want to see the entry.

Would genuinely value critique of the approach.

u/Naive_Maybe6984 — 12 days ago