u/Lumethys

A bit of context: My current company is a team of about 30 people, no dedicated DevOps team, a subsidiary company of a bigger corp.

We have a dozen monolithic codebase on AWS infra, mostly on ECS, a few more arriving on Fargate and Lambda

Lately there's a quite a few instances of legacy bad architectures and coding practices leading to some services essentially DDoS-ing themselves or adjacent dependencies. Coupled with alarming numbers of supply chain attacks and vulnerability recently. Corporate had grow paranoid enough to invest seriously on "monitoring and security enhancement".

I have been advocating for better observability for quite sometimes, but it just stopped at better logging practices and adopting sentry for a couple projects.

This is a golden opportunity to build and pioneer an observability stack, "the right way", and I intend to take every advantages.

My colleagues arent familiar with observability at all, but are willing to learn and adopt better tooling and practices.

As for myself, I have had luck with OTeL + VictoriaMetrics/VictoriaLogs/VictoriaTraces + Grafana for some of my personal stuff. But obviously not on the same scale as ~10 production applications

If it was up to me, I would just use that same stack, but to present a fair overview of the ecosystems for my colleagues and managements, I need to also consider other competitors, like clickhouse-based products like SigNoz, ClickStack,... (and OpenObserve?), as well as third-party vendors like datadog, splunk,...

Documentations and videos could only get me so far, there are a few points that would require extension experience:

1/ Functionality-wise, what could Clickhouse-based products and third-party vendor offer that was not possible on a LGMT stacks or Victoria stacks?

2/ Cost-wise, how would each differs, LGMT vs Clickhouse vs 3rd party? I know this is a very vague questions and depends a lot on specifics, so let just say I have 10 projects that can operate comfortably on a 2vCPU and 8GB RAM ECS instances. How would cost compare?

3/ Strategy-wise. For context, I intend to use the standard Agent-To-Gateway Pattern setup. But should I:

• pick 2 or 3 projects and collect both application and eBPF telemetry?

• collect eBPF telemetry for all projects first and slowly adopt application telemetry, since that would require no code changes for current projects?

• collect application telemetry first and slowly adopt eBPF?

• any other suggestion?

I would loves to hear opinions and experience people has on similar situations

Any insight is appreciated