Should _matchesPosition return bytes or char?

[curquiza]

Following some issues like those ones:

• Chinese _matchesInfo length meilisearch#2149

Currently, _matchesInfo returned by the search engine return integer values in bytes. We might want it to return the value in characters.

We need to choose the default behavior that fits 80% of the usage. If needed, in the future, we may want to expose a settings in Meilisearch to change this behavior.

[miiton]

I think should return char.
This is because UTF-8 has bytes ranged glyphs (1bytes to 4bytes).
I don't really want to think about the implementation of recalculating the character range from bytes on the client side.

[gmourier]

Hi @miiton 👋

Would you still need _matchesInfo if Meilisearch was natively adding a crop marker and let you customize the highlight tags?

[miiton]

Oh, then I don't need it. 👍

[gmourier]

Thanks, @miiton! FYI, we have planned to add it in the v0.27.0 release.

[miiton]

I am reporting an issue that I believe is related to this case.
Sorry if there is already an Issue. (I couldn't find it.)

# launch
docker run --rm -p 7700:7700 -e MEILI_MASTER_KEY=foo getmeili/meilisearch:v1.0.0-rc.1
# create index
curl -H "Authorization: Bearer foo" -H "Content-Type: application/json" -X POST localhost:7700/indexes -d '{"uid":"foo","primaryKey":"id"}'
# add  documents
curl -H "Authorization: Bearer foo" -H "Content-Type: application/json" -X POST localhost:7700/indexes/foo/documents -d '{"id":1,"name":"バナナのチョコレートパフェ"}'
curl -H "Authorization: Bearer foo" -H "Content-Type: application/json" -X POST localhost:7700/indexes/foo/documents -d '{"id":2,"name":"Banana Chocolate Parfait"}'

A search in Japanese returns results that are more than one character over.

curl -H "Authorization: Bearer foo" -H "Content-Type: application/json" -X POST localhost:7700/indexes/foo/search -d '{"q":"バナナ", "attributesToHighlight": ["name"], "showMatchesPosition": true}'

# return:   ..."_formatted":{"id":"1","name":"<em>バナナの</em>のチョコレートパフェ"},"_matchesPosition":{"name":[{"start":0,"length":4}]}}],...
# expected: ..."_formatted":{"id":"1","name":"<em>バナナ</em>のチョコレートパフェ"},"_matchesPosition":{"name":[{"start":0,"length":3}]}}],...
                                                       ^^

English words return the expected result.

curl -H "Authorization: Bearer foo" -H "Content-Type: application/json" -X POST localhost:7700/indexes/foo/search -d '{"q":"banana", "attributesToHighlight": ["name"], "showMatchesPosition": true}'

# return:   ..."_formatted":{"id":"2","name":"<em>Banana</em> Chocolate Parfait"},"_matchesPosition":{"name":[{"start":0,"length":6}]}}],...

It seems similar to @shigma's problem, but I don't know if it is the same.

[shigma]

So is this behavior changed? I tried 1.0.0-rc1 but still get byte length instead of character length (which caused bad highlighting).

[gmourier]

Hello @shigma,

They are still bytes in length; in v0.27, we added features to make cropping and highlighting easier.

Do you have a particular reason to use _matchesPosition?

My concern with this change is that it could impact users with no problem today. Maybe we could do something automatic, depending on the language of the matched field? @ManyTheFish @Kerollmops

[shigma]

Thanks for your quick reply. I use algolia instantsearch in my documentation website, but the highlight isn't correct:

algolia/instantsearch#5444