Protobuf support

[newdev8]

Meilisearch is designed to be fast and efficient, and while that is the case, using protocol buffers instead of JSON would make parsing a bit more faster. I think they could both be an option to choose, saving time and resources for both the server and the end user.

[gmourier]

Hey @newdev8! 👋

Do you currently reach limits because of the JSON format slowing you down?

Thanks 🙇‍♂️

[newdev8]

I am using almost 90% of my power with the current configuration and less than 5k rows, so it won't scale well with the 20k rows we need. I know using protobuf won't make it go down drastically, but I'll help a bit as I'm hosting on a microcontroller. We try to max out our efficiency by using Rust, zero-copy whenever we can, etc... And protobuf would also help.

[gmourier]

Thanks, @newdev8! What are the reasons to use Meilisearch on a microcontroller? Are you restricted to this?

[newdev8]

We use Meilisearch in a microcontroller because the device has to be portable, and to use the least amount of battery possible we chose a microcontroller. Technically we could use other low-end devices (like a Raspberry Pi) but after testing different options, this works best for us as battery can last all day with one charge.

[gmourier]

Here is a tip that can save you indexing and search response time.

Ensure that you define the searchableAttributes to match queries against. By default, Meilisearch is indexing every field to be searchable (the default value is *). So, if you have a lot of informative fields that are not useful to be searched, it could help to remove them by defining your searchableAttributes to gain search/indexing performance and relevancy.

[newdev8]

Thanks for the tip!
We have managed to get around 15ms response time. Very helpful, but still, I think protobuf would also make a significant difference.

[gmourier]

Great, that's quite an enhancement!

FYI, we benchmarked the deserialization performance of JSON with serde a few weeks ago, which was about 500ns on the /search endpoint.

[newdev8]

Nice, although serde is quite fast, it's also about memory consumption. Protobuf can shrink payloads up to 1/5 (in the best cases) compared to JSON.

[newdev8]

Not just talking about RAM, but also network packets.