[Charabia] Feature request: trim emoji during segmentation

[slatian]

Why am I requesting this:

I was tinkering with charabia to get an understanding for it, and even with my "well behaved" small dataset I noticed that people use emoji "like this🤷", which segments to "like" and "this🤷". I'd prefer an option to let it be segmented to "like", "this", "🤷", which would allow multiple things:

• "this" may be a stopword with special treatment, "this🤷" would be a non-obvious bypass to that logic.
• Having all kinds of word variations with just emojis pre-/suffixed seems like a waste of index space and CPU time to query possibly meant variations to me (especially with prefixes).

So my concrete Feature request is an emoji segmenter that segments off prefixed and suffixed emoji clusters into their own tokens.

Related question: Is there a way I could implement this without having to fork Chrabia, I haven't found a way to get a custom segmenter attached to a tokenizer,

[irevoire]

Hey, @slatian, as far as I know, there are no tricks to fix this issue except preprocessing your documents and inserting a space before each emoji. I'm sorry, but it's definitely something we could improve on our side.
I feel like this should be Charabia's default behavior.
However, if we end up deciding that it's not the case, I feel like this issue is actually very much related to this one: https://github.com/orgs/meilisearch/discussions/789
Where we want to separate between two chars instead of defining a separator char.

Currently, most of the team is on holiday; you may have a better answer in a week or so.
Poke @macraig and @ManyTheFish that should be interesting to you

[slatian]

Thanks for the quick answer! I'm definitely interested in what @macraig and/or @ManyTheFish have to say.