How to move from Logs to Traces?
​
Hello everyone.
I built a Centralised Logging Dashboard for my team. Powered by Alloy Collector and Grafana Loki. Now we can comfortably view logs and troubleshoot issues.
Also have a few dashboards powered by LogQL queries.
Now, these aren't sufficient for us. We need traces.
Our Application is built using the ISO20022 Framework. It's a mission critical payments platform.
So the logs have trace ids, message ids, profiler logs...
I understand to get Traces it's best to use Otel for manual application instrumentation so we cen see details of what happened clearly. The challenge is that my team is old-fashioned, so they won't instrument. I am left with Autoinstrumentation using Beyla. Now Beyla can see some things but not enough to build a detailed trace.
This is what I have in mind, a solution that shows me when a request was received, processed in the source station, logged to Kafka, redis interaction...core station, etc...till when response is sent to the requester.
I built a log parser in python, but it isn't fast enough, takes almost 120 seconds to return details of a message ID. Even that is not as detailed as I want it to be.
I need suggestions on how to approach this. To either go from Logs to Traces. Or architecture or tools. Basically ideas to improve our Observability. Free tools actually. Opensource 🙂
Kindly share your insights.