document pre-insert/pre-update transformation

[jkdoshi]

MeiliSearch team - Would it be possible to add a feature where a document gets transformed according to an EL (expression language) just before it gets inserted or updated into MeiliSearch? An example of the EL could be the one used by jq. BTW, jq is also available as a Rust crate.

This would be a "setting" in the index. This would allow one do all kinds of transformations one all incoming documents, without having to mess with the source data.

For example:

• add default values of some optional fields
• flatten/denormalize an incoming object/document
• shrink a document before insertion by dropping fields
  etc.

On a separate but related note, in generalit would be nice to have "hooks" at various points in the system, similar to database "triggers"

[gmourier]

Hi @jkdoshi 👋

This would be a "setting" in the index. This would allow one do all kinds of transformations one all incoming documents, without having to mess with the source data.

We have in mind to add this kind of behavior one day. Initially, the idea is to move towards a plug-in system allowing to apply transformations or to make calls to third parties for translation, speech-to-text, etc. This is not the priority at the moment but it's something that will be seriously thought about at some point.

Today the workaround is to have a transformation pipeline before Meilisearch. How would you imagine this setting?

On a separate but related note, in generalit would be nice to have "hooks" at various points in the system, similar to database "triggers"

Please, could you tell me a little more about what would that allow you to do?

Thanks!

[jkdoshi]

One way to imagine it would be, settings like this (using jq syntax):

"transforms": {
  "pre-insert": ".created = now",
  "pre-update": ".changed = now",
  "pre-save": ".expires = now + 60*60*24*365", // expires in 1 year (for both insert & update)
}

The above set current timestamp to "created" on insert, "changed" on update, and sets the expiration timestamp of the record to 1 year from now on save (for both insert and update). This would apply to every document inserted or updated.