▲ 3 r/devops

Fixing broken Datadog APM gaps

Hi Observability Experts,

I am working with a client managing a large-scale infrastructure across AWS, Azure, and on-premise environments. They have used Datadog for 4–5 years, but legacy technical debt has left their APM configuration fragmented:

The Problem:

APM libraries are out of date, and source-level instrumentations are misconfigured. As a result, end-to-end distributed tracing is broken across key services.

Datadog support confirmed that fixes are required at the source (application code/local agents). However, getting application team bandwidth and stakeholder approval across this many services will take significant time.

We are building a centralized Observability pipeline layer (independent of app teams) to standardize telemetry control, but rollout is months away.

My Question:

Is there a viable temporary/interim solution—such as an intermediate proxy, collector layer, or transformation pipeline—that can intercept, patch, or enrich broken spans/traces in-flight before they reach Datadog’s backend?

Has anyone successfully addressed legacy APM gaps at the pipeline level without waiting for app redeployments?

Thanks for your insights!

P.S. - Datadog support has been already consulted for this and they have recommended to fix it from the code/local agents.

reddit.com
u/JayDee2306 — 3 days ago
▲ 4 r/sre

Datadog APM Remediation - Broken Distributed Tracing

Hi Observability Experts,

I’m working with a client that operates a large-scale estate across AWS, Azure, and on-prem environments. They have been using Datadog for several years, but accumulated technical debt has left their APM implementation fragmented.

Current situation:

  • APM libraries are outdated across multiple services.
  • Trace context propagation and instrumentation are inconsistent.
  • End-to-end distributed tracing is incomplete or broken for many critical applications.
  • Datadog Support has reviewed the setup and confirmed that the proper fix is to update instrumentation and agents at the application/source level.

The challenge is that obtaining application team bandwidth and stakeholder approval across a large number of services will take considerable time.

In parallel, we are planning a centralized observability pipeline layer to standardize telemetry collection and governance, but that initiative is still a few months away from broader adoption.

My question to the community:

Is there a viable temporary/interim solution—such as an intermediate proxy, collector layer, or transformation pipeline—that can intercept, patch, or enrich broken spans/traces in-flight before they reach Datadog’s backend?

Has anyone successfully addressed legacy APM gaps at the pipeline level without waiting for app redeployments?

reddit.com
u/JayDee2306 — 3 days ago

Fixing broken Datadog APM traces

Hi Observability Experts,

I am working with a client managing a large-scale infrastructure across AWS, Azure, and on-premise environments. They have used Datadog for 4–5 years, but legacy technical debt has left their APM configuration fragmented:

The Problem:

APM libraries are out of date, and source-level instrumentations are misconfigured. As a result, end-to-end distributed tracing is broken across key services.

Datadog support confirmed that fixes are required at the source (application code/local agents). However, getting application team bandwidth and stakeholder approval across this many services will take significant time.

We are building a centralized Observability pipeline layer (independent of app teams) to standardize telemetry control, but rollout is months away.

My Question:

Is there a viable temporary/interim solution—such as an intermediate proxy, collector layer, or transformation pipeline—that can intercept, patch, or enrich broken spans/traces in-flight before they reach Datadog’s backend?

Has anyone successfully addressed legacy APM gaps at the pipeline level without waiting for app redeployments?

Thanks for your insights!

P.S. - Datadog support has been already consulted for this and they have recommended to fix it from the code/local agents.

reddit.com
u/JayDee2306 — 3 days ago

ClickHouse POC

Hi all, I'm looking to explore ClickHouse through a personal POC. What would be the best hands-on project to understand its strengths, especially for Observability use cases?

Also, does ClickHouse offer any free trial, learning credits, or evaluation program for individuals interested in trying it?

reddit.com
u/JayDee2306 — 6 days ago

Datadog Admins - Any Recent Wins with AI Automations in Day-to-Day Ops?

Hey observability folks, especially Datadog admins,

I'm curious about real-world automation wins you've achieved using AI recently, particularly around routine Datadog operations.

What AI automations have actually moved the needle in your Datadog operations?

Are you using Claude or Bits AI features (Investigate, Chat)? What's your take?

Any tools beyond Datadog's native offerings that you've found useful?

What still feels too manual that you wish could be automated?

Would love to hear your stories — wins, failures, or lessons learned.

reddit.com
u/JayDee2306 — 1 month ago
▲ 0 r/sre

Datadog Admins - Any Recent Wins with AI Automations in Day-to-Day Ops?

Hey observability folks, especially Datadog admins,

I'm curious about real-world automation wins you've achieved using AI recently, particularly around routine Datadog operations.

What AI automations have actually moved the needle in your Datadog operations?

Are you using Claude or Bits AI features (Investigate, Chat)? What's your take?

Any tools beyond Datadog's native offerings that you've found useful?

What still feels too manual that you wish could be automated?

Would love to hear your stories — wins, failures, or lessons learned.

reddit.com
u/JayDee2306 — 1 month ago

Datadog Admins - Any Recent Wins with AI Automations in Day-to-Day Ops?

Hey observability folks, especially Datadog admins,

I'm curious about real-world automation wins you've achieved using AI recently, particularly around routine Datadog operations.

What AI automations have actually moved the needle in your Datadog operations?

Are you using Claude or Bits AI features (Investigate, Chat)? What's your take?

Any tools beyond Datadog's native offerings that you've found useful?

What still feels too manual that you wish could be automated?

Would love to hear your stories — wins, failures, or lessons learned.

reddit.com
u/JayDee2306 — 1 month ago
▲ 13 r/Observability+1 crossposts

Hi Community,

Has anyone replaced Datadog Agents and/or Datadog Tracers with OpenTelemetry (OTel) Collectors to send telemetry data (metrics, traces, and logs) to Datadog?

For context, Datadog supports ingesting OTel data via the OTLP ingest endpoint, either through the Datadog Agent acting as an OTel Collector or via a standalone OTel Collector forwarding data to Datadog using the Datadog Exporter. This makes it possible to instrument applications using OTel SDKs and send the data to Datadog without relying on Datadog-native agents or tracers.

If you have gone down this path, I would love to understand:

  1. What was your primary motivation for making the switch? (e.g. vendor neutrality, standardization, cost, flexibility)

  2. What did you gain or lose compared to using native Datadog Agents and Tracers?

  3. Were there any limitations or trade-offs you encountered, such as reduced APM feature parity or missing Datadog-specific telemetry?

  4. Would you recommend this approach, and if so, for what type of use cases?

Any insights, experiences, or recommendations from the community would be greatly appreciated!

Thank you!

reddit.com
u/JayDee2306 — 4 months ago