Add support for a config file

[MarinPostma]

Summary

We want a way to configure Meilisearch with config files. The format of the file has not yet been decided upon, but it will be one of JSON, toml, or yaml.

Motivations

This file will serve to consistently launch MeiliSearch instances, as executable options list grows bigger and bigger.

Explanation

The options in the file mirror these already present in the executable options and the environment variables.
We will be able to launch with a configuration file with:

meilisearch -c config.json

or

meilisearch --config config.json

All settings in the config file could be overridden if an option or an environment variable is present for the same setting.

[curquiza]

Great feature!!!

What would be allowed in the config file?

• settings?
• flags? (--no-analytics)
• indexes uid?
• ...

[Kerollmops]

I think we could maybe choose TOML for the same reasons Patrick Walton does for Cargo.

[anborg]

Adding "env override" to specs is helpful.

Override

Override props specific to one instance

meilistart-mysitesearch.sh

#  'predictable pattern' environment variables to override
# 
DB_PATH = /mnt/datastorage/mysite/meili/data/instance1
PORT_BIND = 9191
#CONFIG_FILE=
# config files can be refered also from a http url (github/other)?
/my/bin/meilisearch --config-file file:///path/to/common-config.toml

To put common defaults in a toml file

Encourages org wide standard setting, while individual instance can use.

common-config.toml

#One may have all keys in small letters, hyphenated. 
[search]
db-path = /mnt/disks/datadisk/apps/meilisearch/data/default
db-size = 1 # GB units so it is human readable.
host-bind = 127.0.0.1
#db-update-size = #Perhaps can avoid exposing?
port-bind = 7700

[logging]
log-dir = /var/tmp/logs/apps/meilisearch/default
log-file-pattern=meilidefault.yyyymmdd.log
log-rotation = daily|weekly|monthly ?


[security]
meili-env = production
master-key = test123
ssl-authentication-path = 
ssl-certificates-path=
ssl-key-path =
ssl-ocsp-path = 
ssl-require-auth = true

[additional]
#Perhps may use a uniform "enable" prefix. Perhaps may avoid mixed enable/disable/ no-xx no-sentry etc.
enable-analytics = true
enable-sentry=true
#perhaps if log dirs / data dir is absent this may create. default to false?
create-missing-dirs = false

[gmourier]

Hello @anborg. Thanks for the suggestion, there are some super interesting things.

In which case will you use this feature? Why would it be useful for you? I would like to know more to avoid making assumptions! Thanks 😇

[MarinPostma]

As stated in meilisearch/meilisearch#958, clap (therefore structopt) fails to correctly parse flags when used with an env variable. This force the user to input true to flags to activate them. We need to parse env another way, and merge the config from file/env/args with the correct precedence (and default values).

[gmourier]

Following a refinement session with @Kerollmops and @irevoire.

Constraints

• The chosen format should be human-readable and easy to manipulate.
• The chosen format should allow the configuration to be extended to the level of index settings in the future.
• The chosen format should allow the loading of sub-config files.

Format ideas

• .ini
• TOML
• YAML
• JSON
• cuddle (KDL)
• https://dhall-lang.org/
• HCL

From our exploration, it seems that TOML is a good choice.

• TOML seems easier to edit than YAML and JSON. The indentation of YAML is prone to syntax and validation errors while JSON does not support namespace or comment.
• TOML is less expressive than other formats but Meilisearch does not need as many expression capabilities. We may want to deal with settings per index in the future which is doable in TOML.

Other formats can still be discussed.

Priority order

The ideal order of priority seems to be:

• Configuration file
  • Overrided by ENV file/vars
    • Overrided by CLI options

First iteration

• The default location of the config file is still to be determined.
• This default location must be overridable.
• Accepts all CLI options listed here https://docs.meilisearch.com/learn/configuration/instance_options.html#all-instance-options. See format in next example.

master_key = "thisisasuper1337masterkey@@@"
no_analytics = true
...
ssl_require_auth = true

Bonus points

• Parse and display clear errors message for the configuration scheme. e.g. https://docs.rs/miette/latest/miette/
• Explore how we could have named sections in the config file. See the following example.

master_key = "thisisasuper1337masterkey@@@"
no_analytics = true

[dump]
path = "dumps"

[snapshot]
enabled = true

[ssl]
require_auth = true

[brunoocasali]

+1 TOML

[bidoubiwa]

How does nesting work in TOML?

[curquiza]

What do you mean with nested?

[bidoubiwa]

Imagine some day we have a nested config (I can imagine this wont be the case as it should still be possible to do with flags).

{
  myConfig: {
     mySubConfig: "x"
  }
}

How would this translate in toml?

[curquiza]

So far, no nested configs are planned, if I'm correct. Let me know @gmourier if I'm wrong

In any case, we can use the sections as indicated above

[myConfig]
mySubConfig = true

[curquiza]

@gmourier and @irevoire already made some researches about the best config format file to use: you can see why we chose toml for a first solution in the comment above

However, if some of you (@meilisearch/cloud-team, @meilisearch/integration-team or any Meilisearch users) have a better idea or would expect a different format, please let us know which one and why 😄 the solution is not implemented yet, here see the related issue
Thanks in advance for your participation in this 🙏

[eskombro]

Nonetheless, we would be totally ok if we first develop this feature with TOML support only 🦭

[curquiza]

Thanks a lot @eskombro for your feedback/request. I let @gmourier guide us about what is the priority regarding the first implementation of this feature.
Even if we are not able to implement everything in the first version of the feature, every request is still really appreciated 😄

[irevoire]

Is it common to accept two format for the same config file? (Because then we could also handle the kdl, the best config file format ever made 👌)

[tpayet]

Yes, it is something I've witnessed multiple times as long as the order of priority of read files is explicitly documented ^^ I love that approach that do not force the user to learn a new file format and get to choose his/her preferred format

[gmourier]

This is an interesting idea, to be noted as a future possibility!

At first, I think we can safely converge to the TOML format before adding any other format.

I'm pretty divided on the support of YAML for the moment. There is no lack of examples of frustration with this format for configuration. See this if you want a good laugh 🍍

This is not to knock the YAML format; all formats have pros and cons, but I regularly see a lot of frustrations with the YAML, which should make us not want to throw ourselves at it.

[gmourier]

Hello everyone 👋

Some product updates may interest you regarding that feature request.

---

A first iteration will support the TOML format; it's scheduled to be released for v0.30.

The tracking issue is here.

At first, we had the idea to introduce TOML sections to facilitate the reading of the file, but it imposes some undesired edge effects. See

The simplest solution is to replace the TOML sections with TOML comments for now, to avoid them.

We will also provide a default config.toml listing all the configurations with associated defaults when there are any. The file will contain a comment header to provide a link to the documentation so that the user can quickly learn about the usefulness of the different configurations. See.

The order of priority defined is:

Configuration file < Env variables < command-line options

It means the options in the configuration file can be overwritten by the environment variable set (if they exist), and that can be themself overwritten by the command-line options (if they exist).

[gmourier]

The support for a configuration file is now a reality with the v0.30 release. 🪄

Thanks to it, you can now seamlessly version and share your Meilisearch configurations.

📚 More information here

Locking this discussion. For any feedback about the configuration file support, please open a new one.

Thank you, everyone!