Offline creation of indexes

[angelcervera]

I have an index that is not going to be updated after the creation and it is going to be deployed as a Docker container.

Because it is not going to change after the creation, I want to be able to create the index offline. I mean:

1. Create the index data structure without start the server itself.
2. Create a new a docker image or use the official one but pointing meili_data to the new folder.

On this way, I can make a blue-green deployment everytime I release a new version of the index.

Any suggestion about how to create the index data folder without start the server.

Plan B: If it is not possible, could I start the server, index the data, shutdown the server and copy the meili_data folder into the new Docker image?

[gmourier]

Hi @angelcervera 👋

It's not possible to create the index data without starting the server the very first time.

However, if you keep the exact same version between deployments, you can try different scenarios:

1. Use a snapshot when starting the new Meilisearch container.
   • The snapshot file is bigger than a dump, but it does not reindex anything so it's faster.
2. Use a dump (if you change the Meilisearch version between 2 deployments, this is the way to go).
   • The dump file re-index data. It's used as a migration tool between 2 versions.
3. afaik, if you keep the same version, you should be able to start Meilisearch on the same meili_data folder. @irevoire, do you confirm?

Sorry for the late answer 🙇‍♂️

Please let us know if it helps!

[angelcervera]

Hi @gmourier
I had a conversation in Discord with few people about it. As a work around, I'm starting a new empty server, index data there and then shutdown it when all indexation is complete (I mean, all tasks as been finished successfully)
After that, I create a new Docker image using with the generated data folder inside.

The work around looks working, but would be great to be able to do it without staring the server in the same place.

[gmourier]

Thank you for the follow-up @angelcervera

How often do you do this process? Have you been able to automate it?

I would like to evaluate how critical this is for you today. Thank you for your answers in advance if you have the time to devote to them 🙇‍♂️

[angelcervera]

At the moment, we are regenerating the full index weekly. It is not big index (around 70K records).
The automation process has been simple, and it is running as a GitHub Action every week or when we update something in the index content (like changes in the document schema indexed).

I added to my wish list to write a PoC to try to decouple the part that generate the index from the part that query it, to be able to use it as a library. But TBH, no idea when I will have time to it.

FYI: The idea is the same used in Solr offline batch indexing.

[gmourier]

Thank you @angelcervera!

Did you know that you can use the swap indexes feature to deploy a new index version without downtimes (thinking about your schema changes and weekly re-indexing)

So every week or when a document schema changes the process would be the following:

• indexA is serving search request
• Your automation create a new version of that index under indexB
• When the 70k docs are indexed on indexB you swap the content from indexB to indexA
• If everything is ok you can safely delete indexB and you are good to go

[angelcervera]

Thanks @gmourier for the hint.
I didn't know about this option, but I don't want to have the server running all time.
I'm creating a Docker image with the new index because the idea is to deploy it as a Google Cloud Function or Google Cloud Run, mainly to allow auto-scaling. And more important, scale to zero when it is not used.

I was testing it (still not in production) and the cold start is good enough for our use case. We are in a really early stage of the product and at the moment, saving money is priority one.