Ability to select tokenizer/language

[mooncake4132]

First of all, thank you for this great product. I currently run an in-house search engine and melisearch is way better than mine in almost every aspect. There is, however, a small limitation that is preventing me from adopting meilisearch, and I'm wondering if this is something your team is interested in addressing.

I run my search engine for book and movie titles in Chinese and Japanese. I need a way to force meilisearch to tokenize these titles by characters and not try to guess the tokenizations with libraries like jieba. I have four reasons:

1. Some of the titles get creative. They make up phrases and cause incorrect tokenizations.
2. Titles can be abbreviated into characters taken from the middle of phrases. For example, I'm unable to find 金曜日の妻たちへ with 金妻, probably because the title is tokenized as 金曜日/の/妻/たち/へ.
3. Chinese and Japanese share a lot of characters. It's very possible for the language detector to make a mistake and use the wrong tokenizer.
4. I also want to make the romanizations of titles searchable. They look like latin-based language but shouldn't be tokenized like one -- they may not have any spaces!

All these problems can be solved if I can force meilisearch to tokenize by characters. While these may be specific to my use case, I feel like this is a very useful fallback if the CJK tokenizers don't provide good results. I'm happy to hear what you think!

[gmourier]

Hi @mooncake4132! 👋

We think to open the configuration of tokenizers in the next versions of the tokenizer.

If I understand well you would like to define a general tokenizer able to tokenize in Chinese/Japanese/Latin according to the character?

Do your attributes share multiple languages or can they be separated by language within a document? (having dedicated fields per language, e.g. title_cn, title_jp, ...

I take the opportunity to ping @ManyTheFish who recently worked on the redesign of the tokenizer.

Thanks!

[mooncake4132]

Yes, I would like to have a tokenizer that tokenizes each character into its own term/word, and be able to select this.

Right now my documents look like

[
  {
    "name": "テスト",
    "language": "ja"
  },
  {
    "name": "測試",
    "language": "zh-tw"
  }
]

but I can transform it to whichever format that works best for meilisearch.

Thanks!

[ManyTheFish]

Hey @mooncake4132 and @gmourier!
This use case is interesting, I don't have any solution now, but I will take this in account in future development.
For now, we don't support Japanese texts, this will be released in v0.27 ☺️,
but I'm interested know if Japanese partially fix this issue. 🤔

About Romanization, if you do it, I recommend to just split your romanized words with spaces.
I think meilisearch will handle it 🙂

[mooncake4132]

Thanks for the reply! So it's not that the Japanese text isn't correctly tokenized, it's that they are not supported. A special tokenizer can Japanese can definitely improve my search results -- I look forward to trying it after there is a release candidate for v0.27. Also good point on adding spaces for the romanizations, I will give that a try.