rythm: fix ingestion slack time range

[javiermolinar]

What this PR does:
It fixes the ingestion slack by calculating it according to the partition's last commit.

Using a fixed time on WAL creation won't work as we want for several reasons. Every partition can have a different offset and the wall is created before reading Kafka. Also, we don't have a way to know this start time in case of wal replay.

The adjustment of the ingestion slack should be done outside the WAL, since we could have different strategies. Another possibility could be storing a function to calculate it but I have opted for this approach that extends the current interface without modifying it.

[joe-elliott]

I have not dug into the details of this PR but to share some history about ingestion slack. When we first rolled Tempo out internally it would mark a block's start and time using the min start time and max end time of all spans in the block. We quickly found that every block had a start time of 0 and an end time 100 years in the future due to the data we were consuming. So I created the ingestion slack to prevent this.

Ingestion slack is gross to calculate and I have since just wished I had added 5 minutes to the beginning and end of every block instead of trying to watch all spans and only doing it conditionally.

You all are welcome to tackle this problem however you want, and I'd be glad to discuss it sync if you're like to, but just wanted to give some history.

[mapno]

I like the new behaviour, but I'm unsure of changing how Append and AppendTrace work like that. Specially only changing it for one parquet version. There is no warning to the callers of the functions.