metrics-generator causing a memory leak

[golan118a]

hi everyone
I have an observability stack running on docker and i am using Tempo for tracing.
I am using the monolithic distribution.
My problem is when one of my services is active it generates traces correctly and sends it to Tempo but I notice a massive increase in RAM and CPU. While trouble shooting I have managed to confirm that when I disable the metrics-generator, tempo still runs correctly and the resource abuse does not repeat it self. I still need the metrics-generator for span metrics and service graph.
I would have appreciate any help solving this issue please

yaml files.zip

[joe-elliott]

You have quite a few dimensions defined in your config. My guess is you are generating a very large number of series. I would check the value of the metric tempo_metrics_generator_registry_active_series to see how many series you're generating and remove some of the less valuable dimensions in your config.

[golan118a]

Hi Joe, thank you for the reply

From prior testing I can confirm the increase of memory is due to the dimensions block in both span-metrics and service-graph processors.

(tempo_metrics_generator_registry_active_series - idle: ~300k, at the 30 minutes traces are being sent: ~2M)

If you may, I have some follow up questions:

1. What are the number of dimensions that are recommended to add?
2. As I am not the one who added them, what are the reasons for adding dimensions? Are there any recommended ones to add?
3. The memory use massive size aside, after 30 minutes of new traces being generated and sent to Tempo, I don't understand why does it take so long for the system to release the memory? I can see after 7 hours pulses some of the memory is getting free. Is there a way to fix this in the Tempo yaml or is it something in the golang I cant control?

Thank you again for the help

[joe-elliott]

What are the number of dimensions that are recommended to add?

The number of dimensions matters less than the cardinality. You want to avoid dimensions like IDs and stick to dimensions like http status code and http route.

As I am not the one who added them, what are the reasons for adding dimensions? Are there any recommended ones to add?

The reason to add them is so that span metrics can be broken down by these dimensions in Prometheus.

The memory use massive size aside, after 30 minutes of new traces being generated and sent to Tempo, I don't understand why does it take so long for the system to release the memory? I can see after 7 hours pulses some of the memory is getting free. Is there a way to fix this in the Tempo yaml or is it something in the golang I cant control?

Are you saying you see memory impact 7 hours after no longer sending traces due to metrics? The go runtime will hold on to memory it's not using from an OS perspective as long as there's not memory pressure. This is common for garbage collected languages. Perhaps you're seeing the impact of this behavior?