Config File Format

[maha-hajja]

We're working on a new feature for Conduit, which would able you to define pipelines in a Cofig file, and Conduit will provision these pipelines for you in an immutable state.
This is our first Config file format version, Let us know what you think about it? any feedback, reviews, or questions are all welcomed!

[lovromazgon]

I have a proposal for simplifying the schema. Right now it mostly follows the same structure as the JSON objects we send in HTTP requests. While I think it's a nice thing, I think we could simplify it by removing the config fields.

In the HTTP endpoints config fields were designed this way so that we can reuse the same type in PUT requests, as the fields inside of config are the only mutable parts of an entity. In YAML all parts are mutable, if the config file changes Conduit will make sure that the created entity is exactly as defined, if needed it will drop the old one and create a new one. So the reasoning for a separate config field is not valid anymore. Additionally, it makes the schema more complicated since users need to think about which fields go inside of config and which don't.

Therefore I propose removing the config field and flattening it into the parent entity.

To illustrate, instead of having this:

 pipelines:
   pipeline1:
     status: running
-    config:
       name: pipeline1
       description: desc1
     connectors:
       con1:
         type: source
         plugin: builtin:s3
-        config:
           name: s3-source
           settings:
             aws.region: us-east-1
             aws.bucket: my-bucket
     processors:
       pipeline1proc1:
         type: js
-        config:
           settings:
             script: 'func process() { ... }'

We would have this:

pipelines:
  pipeline1:
    status: running
    name: pipeline1
    description: desc1
    connectors:
      con1:
        type: source
        plugin: builtin:s3
        name: s3-source
        settings:
          aws.region: us-east-1
          aws.bucket: my-bucket
    processors:
      pipeline1proc1:
        type: js
        settings:
          script: 'func process() { ... }'

[maha-hajja]

I agree with this! so much simpler and easier for the user too

[lovromazgon]

We changed the format in Conduit 0.6, read more about the new format: https://conduit.io/docs/pipeline-configuration-files/specifications