nbHits behavior

[curquiza]

I open this discussion to try to make clearer the nbHits behavior that is not always clear for our users.

Meilisearch is a search engine designed for performance and relevancy, not designed to retrieve ALL the matches for a search: only the most relevant documents regarding the user request (a limit of 20 documents by default) will be retrieved.

nbHits returned by the search engine is not the exact total number of candidates, but an estimation of this number of candidates.

Explanation

I will try to explain it in a simple way:
Let's imagine you do this query:

{
  "q": "women"
}

This query means: I ask for the 20 most relevant documents (offset: 0 and limit: 20), so documents from offset 0 to 19.
Meilisearch will first try to find documents matching exactly "women", I mean without any typo. Let's imagine Meilisearch (internally) finds 45 candidates (i.e 45 documents containing "women"): it will apply an optimal sort to return the 20 most relevant candidates (in hits). Also, Meilisearch will return nbHits: 45 because it finds out 45 candidates when applying your request.

Now, with the same database of documents, let's imagine you do this query, for example when doing pagination.

{
  "q": "women",
  "offset": 51
}

This query means: I ask for the 70 most relevant documents (offset: 51 and limit: 20, so 70 documents), and I want Meilisearch to return the document from offset 51 to 70.
Meilisearch will try to get documents matching exactly "women" without any typo first, but in this case, the number of candidates is not enough (45) since you asked for 70. In this case, Meilisearch has to fetch other candidates. How? It will try to fetch documents containing "women" with a typo, like "woman".
Let's say in this case, Meilisearch finds (internally) 100 new candidates with "woman": it will apply an optimal sort to gt the 70 most relevant candidates and return the ones from 51 to 70. Also, Meilisearch will return a new nbHits: 145 because it finds out 100 + 45 candidates when applying your request.

Conclusion

nbHits is a good way to know if there are still candidates matching your query you have not browsed yet, but it cannot guarantee this number is the exhaustive number of candidates.
With nbHits you can not directly access the LAST page of your pagination, but only know if there are next pages. Reaching the last page is not recommended anyway, for performance reasons, because enforcing Meilisearch to get ALL the relevant documents, sort them, to retrieve the latest ones you asked for.

---

Knowing this we are sorry for the potential confusion, and we hope these explanation are clearer. We want to be transparent with our users.
FYI, here are the next steps we are considering and working on:

• the documentation team is working on providing guides to help the users develop the best pagination according to their needs
• we plan to rename nbHits to remove the confusion due to the name in v0.28.0 (planned to July): Confusing nbHits #394
• most of all, the product team is available to get user feedback on their needs regarding the pagination. Maybe, you as a user, absolutely NEED to access the last page, so to know the total and the exhaustive number of candidates for a query. Please let us know about your use case in this discussion! It would be really helpful ❤️

[bb]

Does this mean that as long as I'm not searching but only filtering (facets or geo), I actually can expect nbHits to be exact?

I understand that, as a work around, I can set the limit to a number larger than the total document count (so there's only one page and it will be the first and the last at the same time). Question is, do I need this workaround when only filtering, not searching?

One use case is https://www.mapmf.org/explorer where the numbers on the map and the charts are based on nbHits and/or facetDistribution and these are actually quoted in newspapers etc.

[Kerollmops]

Does this mean that as long as I'm not searching but only filtering (facets or geo), I actually can expect nbHits to be exact?

Indeed, the candidates are computed before the search is performed. However, keep in mind that it can change in the future, we do not guarantee that we will not change this behavior. For now, you can rely on it.

I understand that, as a work around, I can set the limit to a number larger than the total document count (so there's only one page and it will be the first and the last at the same time). Question is, do I need this workaround when only filtering, not searching?

You don't need this workaround for when you do not search but only use filters, at least for now.

[curquiza]

Update on this discussion: we've just created this discussion to get your feedback about the Meilisearch end-user pagination in general, not especially related to nbHits.
I also updated the description of this discussion accordingly

[gmourier]

Locked in favor of #483