Weird korean tokenizer behavior

[intoxicated]

is this normal behavior for korean tokenizer?

감 shows two results and 감독 shows one result where previous two records contain 감독, (highlighting is also inaccurate)

[ManyTheFish]

Hello @intoxicated,

Thank you for the report, about this highlighting issue, could you retry with version v1.0.2? This should be fixed. If it's not the case, don't hesitate to open a bug report issue directly on Meilisearch pinging me.

Regarding Korean support, we recently changed how we segment words in this PR: meilisearch/charabia#154.
We have a dedicated discussion for Korean Language support: #564,
As a Korean speaker, you may help me target this change's limits and suggest modifications to enhance the Korean Language support!

I lock this conversation in favor of the dedicated discussion below:
#564

Thanks again!