Fetch all search results page by page in a performant way / auto-increment ID?

[underflow00]

I need a way to fetch all search results page by page in a performant way.
My understanding is that using offset is not performant as it requires fetching and processing the skipped documents. Is that correct?

I've thought of adding a unique filterable number attribute and using that as a cursor for pagination (filter = "id > {lastId}"). Is that a good idea? Is there an easier way, given that there is no built-in auto-increment ID (old proposal here) and I would have to maintain this ID in a separate database?

[irevoire]

Hey @underflow00,

My understanding is that using offset is not performant as it requires fetching and processing the skipped documents. Is that correct?

No! Using the offset, even without a filter, should be very performant; you shouldn't have to do any trick.
Internally, Meilisearch is already assigning IDs to your documents and then storing them in a RoaringBitmap.
That's a special kind of data structure that allows us to very quickly skip tons of IDs to reach the point we're looking for.

Did you notice any performance issues?