Tempo querier sends a lot of request and got 206 response code

[zhouyanbl]

Hello,
We have observed that the Tempo querier component is sending a large number of requests — approximately 1,500 queries per second (qps) — using the GET method, which results in a 206 response code. However, the request rate from users on the Grafana Tempo UI is about 6 to 20 qps, which seems insufficient to explain the high volume of requests generated by the Tempo querier.

We have some main questions:

Why there is so many requests like it?
What exactly is the querier component accessing? Are these requests being sent to Amazon S3? If so, this could potentially lead to significant S3 costs.
How can we debug the querier to understand what it is specifically requesting?
Thank you for your assistance.

[joe-elliott]

This is possibly due to polling. A few questions that will help me understand what may be going on:

How many blocks do you have?
What version of Tempo are you running?
Do you have cache setup?

[zhouyanbl]

Good day!
1.Our system generates a block approximately every 1-5 hours, and each block is retained for 90 days.The logs corresponding to our queries within a short period are closely timed, with queries incorporating a 120-second time limit. The queried blocks are concentrated within a specific time frame, according to log entries like "searching trace in block".And some of data.parquet can be several GBs.

2.We forked the repository on November 17th. The version we are currently running is between 2.3.0 and 2.3.1.
3.We do not have caching set up. According to the operations team, attempts to implement caching in the past did not produce significant results.
Thank you!

[joe-elliott]

1.Our system generates a block approximately every 1-5 hours, and each block is retained for 90 days...

tempodb_blocklist_length will tell you this exactly. 90 days is quite long. You may need to increase compactors.

2.We forked the repository on November 17th. The version we are currently running is between 2.3.0 and 2.3.1.

I would recommend updating to 2.4. We made some large improvements to polling to reduce calls to the backend.

3.We do not have caching set up. According to the operations team, attempts to implement caching in the past did not produce significant results.

2.4 also came with some very nice caching improvements. If you continue to see high rates after updating to 2.4 then I'd consider caching footers and blooms.

See "tiered caching" and "Polling and TCO Improvements" sections:
https://grafana.com/blog/2024/02/29/grafana-tempo-2.4-release-traceql-metrics-tiered-caching-and-tco-improvements/

[zhouyanbl]

HI bro,we upgraded it to v2.4 and it worked well now.Thank you for your advice!