service graph is not showing average time correctly

[maheddin092]

Hey,

I am implementing traces in my application, here the connection type is "messaging" and the application is using Kafka for communication.
In my application, I am consuming from Kafka inserting the message to an internal queue pulling the message in another thread and processing it, and then sending it to another external app that is not instrumented. I am getting the trace details in Grafana, but is the service graph average time calculated by considering only root span time, can any one help me here to fix this.

LABLES:
consuming:

processing from que:

sending

service graph:

trace:

here the average time should be 3490 ms, it is only considering the root span time which is 1 sec.

[joe-elliott]

I believe this is just showing a "service graph" of the entry point into your trace. Asynchronous service graph creation is disabled by default but you can turn it on by setting enable_messaging_system_latency_histogram to true under the service graph processor block on the metrics generator config map:

https://grafana.com/docs/tempo/latest/configuration/#metrics-generator

We don't use this setting internally. It was PR'ed by the community. I would love to hear how it works for you.

[maheddin092]

Hi,

Even after configuring the results are the same.

Configuration:

service graph:

trace :

Is there any link( context link) that has to be done? between the receive and process span (pulling from the Que) or any labeling I am doing wrong.

I will be very thankful for your reply

[joe-elliott]

We don't use this feature, but I assume it works. Are the span kinds correcly labeled as Producer and Consumer?

Here's a metrics generator trouble shooting guide:

https://grafana.com/docs/tempo/latest/troubleshooting/metrics-generator/

[nic-godz]

I'm not sure I follow...but is the issue about the "service graph" showing incorrect average and the "trace graph" showing correct?

Have you tried prolong the lifetime of the pCL_OtelTraceRoot object for entirety of the underlying span A & B objects? Just to make sure how this will be presented in the "service graph".