--max-index-size and --max-task-db-size are not practical

[curquiza]

As a user, to control the space Meilisearch will take on my disk, I would love to have only one configuration to set this instead of two, like a --max-db-size, including the task database AND the indexes database.

Also, if we don't have the choice and I have to set 2 DB sizes, I would rather set the max size of the database for ALL my indexes, and not per index. For me, --max-index-size is not practical.

I try to think as a user, I don't say this is easy technically, but I want to reconsider these both commands we provide to the users, which are not really practical and intuitive from my POV.

[gmourier]

Sharing @nicolasvienot's input

Currently, we set both to the size of the disk we use:

# For a disk of 100Gb
MEILI_MAX_INDEX_SIZE=99999997952
MEILI_MAX_TASK_DB_SIZE=99999997952

Of course, this does not really make sense, but we mostly wanted to start using those variables in order to someday be able to really limit the size of the Meilisearch DBs to the size of our disk.

We can’t really use those variables as it would mean:

1. Define what % of disk we want to allow to indexes and what % we want to allow to tasks
2. Redefine the MEILI_MAX_INDEX_SIZE depending on the number of indexes

When implementing that, yes we were hoping for a way to actually set a disk limit for Meilisearch. So a variable that could let us just set the max size for the whole Meilisearch would be a lot better for our use case.

Although there might be other use cases where a user wants to limit the size of a specific index, a way to set a limitation for all indexes would be also nice. Since there is no way to separate the location of the tasks and the indexes data, I don’t think we really need the MEILI_MAX_TASK_DB_SIZE for now.

To recapitulate, for our use case, a MEILI_MAX_GLOBAL_SIZE is more interesting than having different variables and a MEILI_MAX_INDEXES_SIZE is more interesting than a MEILI_MAX_INDEX_SIZE.

[curquiza]

So a variable that could let us just set the max size for the whole Meilisearch would be a lot better for our use case.

@meilisearch/engine-team how complicated would it be to implement this?

[gmourier]

I've drawn some graphs to measure the usage of these configurations, we're talking about a usage rate below 1% among all the instances sharing the telemetry infos.

[Kerollmops]

After a small meeting with @dureuill, @irevoire, and @gmourier, we defined that this is not a simple task to do. The current system internally uses the LMDB max size functions to specify the size of the indexes, one by one. We would have to design a smart algorithm to fairly distribute the disk usage by index. Note that changing the max map size of the env requires closing it and blocking read and write operations.

I would also like to note that the current implementation of the index-scheduler doesn't store the update files e.g. JSON documents, before being processed, in the LMDB env, and therefore the --max-task-db-size doesn't work as a user would expect.

We will continue the discussion about exposing a much simpler-to-use flag in the future.

[gmourier]

Thanks, @Kerollmops I think we'll have to talk about it again soon (probably next week); a topic we'd like to explore with the cloud team before v1.0 might lead to that discussion.

[dureuill]

So, we started preliminary work on this topic, and I would like to raise various points:

Do we want a MEILI_MAX_GLOBAL_SIZE?

It is unclear to me that using a configuration option to limit the disk space taken by a Meilisearch instance is the best location.

Reliably self-limiting the space taken by an application is hard

While it is obviously the most convenient location to users, and so well aligned with Meilisearch's mission, it is also something that is hard to achieve reliably from within an application.
I can think of two main approaches to achieve that:

1. Actively track the space we're using. This is a continuation of the approach of monitoring the size of the indexes but requires summating all the sizes, and not forgetting any file, including temporary and update processing files. I think this is a very error-prone approach that will require a lot of additional work to implement and maintain.
2. Passively check the space we're using by performing the equivalent of a du -s on the Meilisearch directory at runtime, and then blocking new document additions after reaching a limit. The limit will be a soft limit, obviously, meaning if the size raises drastically before two checks we won't be able to block that. Also, I'm unsure how it would work performance-wise because we would have to repeatedly make system calls to get the updated size of all files in the Meilisearch directory, which can grow a lot (hence the issue in the first place).

Operating systems are well-equipped to limit the space allocated to an application

OSes (at least Unix-like ones) already have ways to limit the available space:

• on a file-system
• for a given user
  using quotas

so the function could be achieved externally, without implementing it inside of the Meilisearch binary.

To still be helpful to users, I wonder if we could not provide a guide on how to set up a Meilisearch instance with a global quota. What do you think?

So, to summarize my proposal here:

1. I would suggest we do not add a MEILI_MAX_GLOBAL_SIZE or equivalent option to Meilisearch v1.
2. I suggest we instead (optionally) document the use case of limiting the disk space available to one Meilisearch instance if this is important enough a use case.

[dureuill]

What to do of --max-index-size and --max-task-db wrt v1

Regardless of what we decide about a MEILI_MAX_GLOBAL_SIZE option, we need to decide what to do with the existing options (--max-index-size and --max-task-db) before the stabilization of v1. My current inclination is to remove the existing options. Here's why:

1. These options, as seen by the current open discussion, confuse users who would like to use them as a MEILI_MAX_GLOBAL_SIZE option, whereas this is not their use case.
2. --max-task-db ignores the update files associated with the tasks (because they are not directly stored in the index that stores the task queue). During processing, the update files may typically represent the bigger part of the task queue, disk space-wise.
3. This is somewhat easier to remedy, but I think we are missing documentation about what happens when the limit is reached, like what kind of errors a user will receive, and what they can do when this happens.
4. While limiting the size of a specific index is certainly a use case, it is unclear to me that it pulls its weight regarding the points above.
5. While limiting the size of a specific index is certainly a use case, I think it makes sense to be able to limit the size of each index, per index, which means it should probably be a setting for each index.

So, to summarize my proposal here:

1. Remove the --max-index-size and --max-task-db options for v1.

[dureuill]

That's an interesting idea, as it doesn't really raise the complexity by much!

However, am I correct in thinking that:

1. While the original index is copied to a new index with roughly twice the maximum disk space taken, it would mean that for OSes like Windows that physically allocates the maximum index space, we would take roughly x3 disk space?
2. While the original index is being copied and before it is hot-swapped, we could still read from it, but any write would have to stay enqueued (or blocked while processing) due to the original index being full, is that correct? Is copying an index fast?

[Kerollmops]

Indeed, both points are true.

About compacting the database we use the MDB_CP_COMPACT option, here is what the documentation says about it:

Perform compaction while copying: omit free pages and sequentially renumber all pages in output. This option consumes more CPU and runs more slowly than the default.

By closing and reopening the env with a higher map size we avoid the first point of duplicating the data and we also reduce the write downtime by a lot as we don't need to copy the entire database.

We should test only one small thing: Can we detect that we no longer have space when writing into LMDB? I remember that LMDB gives strange error messages like those ones:

• Cannot allocate memory (os error 12).
• MDB_MAP_FULL - Environment mapsize reached.
• StorageFull, maybe grenad is returning that. Or not as it is a nightly category.
• No space left on device (os error 28) or I/O error (os error 5), returned by grenad for sure.

I think we only want to react to the MDB_MAP_FULL error. We will not change the mapsize anyway.
But, we would like the Cloud team to increase the size of the disk by informing them that Meilisearch needs more, right?
I also advise you to read the mdb_env_set_mapsize documentation.

[dureuill]

Here's a summary of the various possible solutions here.

1. No change, keep --max--index-size and --max-task-db-size

Pros

• Nothing to do

Cons

• These options are not practical, as per this current discussion: in particular, they might lead the users to believe they can be used effectively to limit the maximum disk size of a Meilisearch instance, while they practically cannot.
• We'd probably want to move at least --max-index-size to a "per-index setting"
• This stabilizes these options for v1, so we can't remove them afterward.

2. Remove --max--index-size and --max-task-db-size for v1, replacing them with nothing

Pros

• Minimal work
• Don't commit to options we would like to remove afterwards for v1

Cons

When no value is given for these options, they default to a value of 100GiB.

This can be an issue for 3 categories of users:

1. Users using these options to limit the disk space taken by Meilisearch. While these options are not practical, they can still be used for this purpose. I think such users could be redirected to use the "infrastructure-based" solution described in the other reply
2. Users that have small disks and are running Windows (and possibly MacOS?) will have the 100GiB used per user-created index + for the task queue, without a way to make the max size of the index smaller (that becomes more of a problem if we increase the default value of 100GiB).
3. Users that have big datasets and need indexes bigger than 100GiB (this is a problem mostly if we don't increase the default value of 100GiB, but see point (2)).

3. Remove --max--index-size and --max-task-db-size for v1, and have the indexes start with a small size, but grow dynamically when they're almost full

Pros

• Don't commit to options we would like to remove afterwards for v1
• Helps users of categories (2.2) and (2.3)

Cons

• Introduces a lot of additional complexity for v1, with the associated risk of bugs
• We should also have a way to set it so it doesn't grow larger than the remaining disk space, tops. We might want a way to signal to users they need to provision more disk space when reaching this condition. Detecting this condition reliably can be difficult.

4. Remove --max--index-size and --max-task-db-size for v1, and have the indexes start with a small size, but grow dynamically when they're almost full by copying the full index to another index with a bigger max size and then hot swapping the indexes

Pros

• Don't commit to options we would like to remove afterwards for v1
• Helps users of categories (2.2) and (2.3)
• Somewhat easier structurally (so smaller cost to implement and smaller risk) as it reuses already existing features

Cons

• Increase disk space taken during the copy to roughly 3 times the size of the original index (for Windows)
• Does not address what to do when there is insufficient disk space
• Leaves the index unavailable for writing while it is being copied (since it is full), which can take some time

5. Remove --max--index-size and --max-task-db-size for v1, introduce a max-size setting per index

Pros

• Don't commit to options we would like to remove afterward for v1
• Helps users of categories (2.2) and (2.3): they would have to use that option on their indexes.
• Less structural risk?

Cons

• What about the max size of the task db?

[dureuill]

Discussed this with irevoire and Kerollmops:

Seems to us like solution (2), with the twist of using a big default value for the map size (bigger than 100GiB, maybe the total disk size?) is an interesting first step, given that the following open questions are resolved positively:

1. Is Windows still pre-allocating map files for modern-day lmdb? It appears as if MacOS isn't.
2. Is lmdb taking more space than necessary when working with a large mapsize? We would need to benchmark this
3. We need another way to express the size available to soft-deleted documents, as it currently uses the --max-index-size option for this purpose

[dureuill]

Opened an issue about this: meilisearch/meilisearch#3231

[gmourier]

📡 Update 👋

Cross-posting @dureuill comment about the limitations that are brought by this change (removing max-index-size and max-task-db-size).

---

So, after discussing this with the team, we decided to move forward with removing the flags and setting each index size to 500GiB for now.

This change has 2 consequences (besides the fact it is breaking due to removing CLI options):

1. The number of indexes that can exist simultaneously in a Meilisearch DB becomes around 200 for Linux/MacOS and around 20 for Windows. This is due to OS limits on the amount of virtual memory allocatable by a single process.
2. The size of an index cannot grow beyond 500GiB.

We expect at least (1) to affect some users (it is unclear if (2) will affect users in practice, however).

This change is currently implemented in meilisearch/meilisearch#3278.

This change is taken now so as to avoid stabilizing --max-index-size and --max-task-db-size in v1, because we probably don't want them as-is in the future.

We plan to lift the described limitations (1) and (2) in v1.1 (either by adding back index size control as a per-index setting, and/or by abstracting this away completely so a user doesn't have to consider the "number of indexes/maximum size of indexes" tradeoff at all).

---

Knowing this and looking at our analytics, we estimate that a few users will be affected by the limit on the number of possible indexes and the index size.

📚 If you use many indexes to implement a multi-tenant search, you may have missed the tenant tokens feature; It should be helpful to avoid having one index per tenant.

It is possible that the analytics at our disposal do not give us a complete representation of the pool of users that will be impacted by this change; that's why we are already working to solve these limitations. Some tracks are discussed here meilisearch/meilisearch#3280

If you encounter a problem induced by these limitations, please mention it here! 🙏

[vincentsch]

One point that should be noted is that it is not at all possible to estimate how large the index will end up. The index size cannot be determined by the size of the indexed data. It would be very frustrating if you decide to use Meilisearch and then later realize that the index unfortunately exceeds the upper limit.

[dureuill]

Hello @vincentsch, thank you for reaching out to us!

I understand that the new limit of 500GB for indexes would mean that you cannot use the upcoming Meilisearch v1.0. This is unfortunate, and we are currently working on lifting this new limit for v1.1.

To provide some perspective on the new limits, they do have a technical origin.

1. Our database uses memory mapping to let the OS transparently map the contents of the indexes to virtual memory areas.
2. In the current implementation, the size of the mapped virtual memory areas must be known upfront and is fixed for all indexes.
3. Operating systems limit the total size of virtual memory address space available to a process. This value is OS (not hardware) dependent and does not appear to be configurable (at least not to grow it from its base value). If opening an index would mean going above the limit, the operation would fail.
4. Some users are maintaining several indexes in a single Meilisearch instance, and so need to be able to have multiple indexes open.

So there is a tradeoff in the current implementation between the maximum allowed size of indexes and the maximum number of allowed indexes. When you go above that limit you run in issues such as meilisearch/meilisearch#1841 meilisearch/meilisearch#2875 meilisearch/meilisearch#2832 meilisearch/meilisearch#2616 meilisearch/meilisearch#2535.

Prior to v1, you could actually act on this tradeoff using the --max-index-size flag (which I think you must currently be using as its base value is 100GiB). The decision to remove this option is motivated by the following:

1. As we're releasing v1, CLI options are being stabilized. That means that if --max-index-size is in v1.0, it will remain in v1.x for any value of x.
2. We don't think that --max-index-size is the right way to manage the inherent tradeoff that I described above.
   • The tradeoff is not at all known by users
   • The tradeoff has a very high granularity as you have a single value you can set for all indexes, whereas we could prefer setting the value at the index granularity
   • As you said, it is not at all possible to estimate how large the index will end up upfront.
3. --max-index-size was used in the wild to attempt to limit the maximum size a Meilisearch instance can grow to, but it is also ill-suited to this purpose: this option does not take into account the size of the data that is being indexed, that is significant and stored on disk.

Our goal with the ongoing work for v1.1 is to modify the implementation so that the mapped memory for an index grows as needed with the size of the index, and the number of indexes simultaneously mapped in memory becomes capped, effectively entirely removing the limit on both index size and index count (unless some of your indexes become really big, such as an index approaching 128TB of disk space, in which case we expect other things to break first).

As long as we removed the --max-index-size option, these changes can be made in a non-breaking way, but they're subtle changes that we did not want to be rushed for v1.0.

For choosing the 500GiB value, we used our telemetry data to look at the size of the indexes of our users that use telemetry and found no user with >500GiB indexes. Actually, we expect more users to encounter an issue due to the limited number of indexes since some of the telemetry data indicate a small proportion of users using more than 200 indexes.

I hope this answer sheds some light on the current situation, our recommendation for your case would be to wait for v1.1 to upgrade from v0.30. If this is not possible for you please reach out to us.

Thanks again for your feedback it is very important to us to hear from people affected by this change so we can confirm our priorities. Feel free to answer back in this thread if you have further questions or remarks.

[vincentsch]

@dureuill Thank you for taking the time to provide a detailed explanation. I wasn't aware that the restriction was going to be removed in v1.1. Skipping a version is not a problem.

[dureuill]

You're welcome! We'll do our best so we can remove the restriction in v1.1 as planned 😃

[dureuill]

Hello @vincentsch 👋

I'm coming back to you just to let you know we released a prototype that removes the restriction to 500 GiB on index size. If you're interested, you can help us make sure that the prototype correctly fixes your problem without introducing any new issues.

You can grab it here: meilisearch/meilisearch#3382 (comment)

[thijndehaas]

My --max-index-size is set to 250GiB because my index with all website products is 131G. I have 1.5T of disk space so 250G is not a problem but I'm not sure how I can resolve this without removing data that improves the search results or are used in product filters. The database currently contains 2.600.000 products but this will likely increase in the coming years.

For example the website on the primary search page, it is not slow with this size:
https://winkelen.nl/zoeken

[dureuill]

Hello @thijndehaas, thank you for your message!

The plan for v1.0.0 is the following:

• Remove --max-index-size
• Use an internal, fixed value of --max-index-size of 500GiB.

Do you expect this plan to affect your use case? If so, I'd love to hear more from you and how you think this might cause an issue!

[thijndehaas]

Yes, our largest operating Meilisearch website is 131GB but we are planning on migrating larger systems and on a dedicated server this still runs very smooth. So at the moment we don't need it but as @vincentsch mentioned other people will and I'm wondering if there are any negative side effects when the performance is still great.

[dureuill]

I laid out the technical side effects of opening multiple large indexes that motivated our decision in a comment in @vincentsch 's thread.

The plan is to remove limitations on index size and index count in v1.1. This is already being worked on 😃

[gmourier]

Hello, everyone following this discussion 👋

We have just released the first RC (release candidate) of Meilisearch, containing the removal of limits on index size and number!

You can test it by using

• the release assets
• the Meilisearch Docker image

docker run -it --rm -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:v1.1.0-rc.0

You are welcome to communicate any feedback about this new implementation in this discussion.

If you encounter any bugs, please report them here.
Thanks in advance for your help and your involvement in Meilisearch ❤️

🎉 Official and stable release containing this change will be available on 3rd April 2023

⚠️ RC (release candidates) are not recommended for production

[gmourier]

Hey folks ✨

The limit introduced in v1.0 when removing the --max-index-size and max-task-db-size have been lifted with v1.1

Meilisearch no longer limits the size nor the amount of indexes a single instance may contain. You can now create an unlimited number of indexes, whose maximum size will be only dictated by the memory address space devoted to a single process by the OS. Under Linux, the limit is at about 80TiB.

Note that the task db is limited to 10gb, we are working on solving a case where reaching that limit block write operation on Meilisearch. In the meantime, we recommend regularly deleting finished tasks to avoid that if you don't need to keep the finished task historic.