Variant 2

This repository contains a development branch of the second major version of Variant.

See https://github.com/mumoshu/variant for more information on the first version.

Once finished, this repository will eventually take over the master branch of the original variant repository.

Features

• HCL-based DSL: Terraform-like strongly-typed DSL on top of HCL to define your command. See Configuration Language below.
• Concurrency and Workflow: Embedded workflow engine with concurrency. See Concurrency below. Example: concurrency
• Auto-prompt: Variant prompts for missing arguments to your command so that your user is only needed to provide missing arguments on the fly, instead of rerunning the whole command by repeating every argument.
• Configs: Deep-merging YAML configuration files. Example: config
• Secrets: Deep-merging secret values from Vault, AWS SecretsManager, SOPS, etc. powered by vals. Example: secret
• Testing: Test framework with go test-compatible test runner. Example: simple
• Embeddable: Easy embedding in any Golang application
• Easy distribution: Build a single-executable of your command with Golang
• Dependency Management: Dependent files, executable binaries and docker-run shims can be automatically installed and updated with the variantdev/mod integration. Example: module
• Run as a Kubernetes controller: You can easily turn your Variant command into a Kubernetes controller. See examples/controller
• Integrations: Integrates nicely with Slack, GitHub. You can run Variant command in response to Slack message, GitHub issue comment, commit push, etc.

Getting Started

Create an variant file that contains the following:

examples/getting-started/getting-started.variant:

option "namespace" {
  description = "Namespace to interact with"
  type = string
  default = "default"
  short = "n"
}

job "kubectl" {
  parameter "dir" {
    type = string
  }

  exec {
    command = "kubectl"
    args = ["-n", opt.namespace, "-f", param.dir]
  }
}

job "helm" {
  parameter "release" {
    type = string
  }
  parameter "chart" {
    type = string
  }
  option "values" {
    type = list(string)
  }

  exec {
    command = "helm"
    args = ["upgrade", "--install", "-n", opt.namespace, param.release, param.chart]
  }
}

job "deploy" {
  description = "Deploys our application and the infrastructure onto the K8s cluster"

  step "deploy infra" {
    run "helm" {
      release = "app1"
      chart = "app1"
    }
  }

  step "deploy apps" {
    run "kubectl" {
      dir = "deploy/environments/${opt.env}/manifests"
    }
  }
}

Now you can run it with variant:

variant run -h will show you that all the jobs are available via sub-commands:

$ variant run -h

Usage:
  variant run [flags]
  variant run [command]

Available Commands:
  deploy
  helm
  kubectl

Flags:
  -h, --help               help for run
  -n, --namespace string   Namespace to interact with

Use "variant run [command] --help" for more information about a command.

And variant run deploy -h for the usage for the specific job = sub-command named deploy:

Deploys our application and the infrastructure onto the K8s cluster

Usage:
  variant run deploy [flags]

Flags:
  -h, --help   help for deploy

Global Flags:
  -n, --namespace string   Namespace to interact with

As you've seen in the help output, variant run deploy runs the deploy job, which in turn runs kubectl and helm to install your apps onto the K8s cluster:

$ variant run deploy

Once you're finished developing the command, let's build a single executable binary of the command for easy distribution:

$ variant export binary ./ build/myapp

The exported executable binary accepts the same arguments as variant run:

$ ./build/myapp -h

$ ./build/myapp run deploy

Congratulations! You're now ready to dive deep and solve your own problems with Variant.

Still curious how Variant helps developing your own command as it grows?

Head over to the following per-topic sections for more features:

• Writing Commands to learn the Variant DSL for writing commands
• Debugging Commands to learn how to debug your Variant command
• Writing Tests to learn the Variant DSL for writing tests
• Generating Shims to make your Variant command look native
• Compiling Command to export Go source or an executable binary
• Running Command From Other Directory
• Concurrency section to make kubectl and helm concurrent so that the installation time becomes minimal
• Log Collection to filter and forward log of commands and the arguments passed to them along with their outputs
• Use Split, Merge and Import to split, compose and tidy Variant commands
• JSON Configuration Syntax can be used as an alternative to HCL2-based one
• Slack integration to turn your command into a Slack bot

Generating Shims

If you're distributing this command with your teammates, do use variant generate shim to create a shim to make it look like a native command:

$ variant generate shim examples/getting-started/

$ cat ./examples/getting-started/getting-started
#!/usr/bin/env variant

import = "."

$ ./examples/getting-started/getting-started
Usage:
  getting-started [flags]
  getting-started [command]

Available Commands:
  deploy      Deploys our application and the infrastructure onto the K8s cluster
  helm
  help        Help about any command
  kubectl

Flags:
  -h, --help               help for getting-started
  -n, --namespace string   Namespace to interact with

Use "getting-started [command] --help" for more information about a command.

Compiling Command

As we've covered in the Getting Started guide, variant export sub-commands can be used to export your command in various formats.

variant export binary SRC CMD generates an executable binary at the path CMD from your command defined under the directory SRC:

$ variant export binary ./ build/myapp

The exported executable binary accepts the same arguments as variant run. So myapp -h corresponds to cd $SRC; variant run -h, while myapp run deploy corresponds to cd $SRC; variant run deploy.

variant export go SRC/CMD PKG generates a directory at PKG/CMD that contains Go source files that can be built by running go build PKG/CMD.

Assuming you already have go installed, you can run variint export go src/myapp build, then edit code under build/myapp to make any customization that can't be done with shims, and finally build an executable with go build -o myapp ./build/myapp.

Running Command From Other Directory

Usually, when your command has been defined under the directory path/to/your/command, variant run requires you to chdir to it before running.

To be clear, cd $SRC; variant run can be used to run it from another directory. However with that you command cannot access the actual current directory, as you've already cded.

You can also use a shim or an exported binary to make it runnable from any directory. But it takes some time so probably you'd like to run it from any directory without a variant export step while developing?

The VARIANT_DIR environment variable might be the solution. When variant recognizes it, it reads the command from the directory specified by it.

Just run:

VARIANT_DIR=$SRC variant run

Your command can now be run without cd and still has access to the current directory.

Split, Merge and Import

Do you have a huge yourcmd.variant that needs to be split for readability?

path/to/yourcmd.variant:

job "foo" {
  # snip
}

job "bar" {
  # snip
}

job "baz" {
  # snip
}

Variant usually works per-directory basis. That is, it loads and merges all the .variant files in a directory to form a single command.

That is, you can just split the file into three .variant files in the same directory to split the huge file:

path/to/yourcmd_foo.variant:

job "foo" {
  parameter "param1" {
    # snip
  }

  # snip
}

path/to/yourcmd_bar.variant:

job "bar" {
  # snip
}

path/to/yourcmd_baz.variant:

job "baz" {
  # snip
}

Okay that works. But you ended up too many files in a single directory?

A "parent" variant file containing import or imports can be used to load all the *.variant files in the directory into the current job.

path/to/yourcmd.variant:

job "foo" {
  import = "./foo"
}

job "bar" {
  import = "./bar"
}

path/to/foo/foo.variant:

parameter "param1" {
  # snip
}

# snip

Note that imports is the newer variant of import that supports multiple sources to be imported.

Also, you can import following sources:

• Relative path to local directory (A local path that doesn't start with /, like foo/bar)
• Absolute path to local directory (An absolute path that starts with /, like /variant/modules/example.com/foo/bar)
• URLs to Git repository (REPO_URL@PATH/TO/DIR?ref=BRANCH, e.g. git::ssh://git@github.com/mumoshu/variant2@examples/advanced/import/foo?ref=master)

See the import example for the full declaration of this command for reference.

Concurrency

The example in the Getting Started guide can be modified by adding needs to build a DAG of steps and concurrency for setting the desired number of concurrency:

BEFORE:

job "deploy" {
  step "deploy infra" {
    run "helm" {
      release = "app1"
      chart = "app1"
    }
  }

  step "deploy apps" {
    run "kubectl" {
      dir = "deploy/environments/${opt.env}/manifests"
    }
  }
}

AFTER:

job "deploy" {
  concurrency = 2

  step "deploy fluentd" {
    run "helm" {
      release = "fluentd"
      chart = "fluentd"
    }
  }

  step "deploy prometheus" {
    run "helm" {
      release = "prometheus"
      chart = "prometheus"
    }
  }

  step "deploy apps" {
    run "kubectl" {
      dir = "deploy/environments/${opt.env}/manifests"
    }
    needs = ["deploy fluentd", "deploy prometheus"]
  }
}

Now, running variant run deploy deploys fluend and prometheus concurrently. Once finished, it deploys your app, as you've declared so in the needs attribute of the run "deploy apps" {} block.

Log Collection

log block(s) placed under a job can be used to forward log of commands and the arguments passed to them along with their outputs.

Let's assume you'd like to collect log for the test job:

job "test" {
  run "echo" {
    message = "foo"
  }
}

Collecting all the run and exec that are executed as part of this job along with all the metadata attached to respective event can be achieved by filtering events by condition and formatting log messages with format like:

log {
  collect {
    condition = event.type == "exec"
    format = "exec=${jsonencode(event.exec)}"
  }

  collect {
    condition = event.type == "run"
    format = "run=${jsonencode(event.run)}"
  }

  # The log file to be created
  file = "${context.sourcedir}/log.txt"
}

In case you need to post-process or upload the log file, use forward to pass the log file to any Variant job:

file = "${context.sourcedir}/log.txt"

forward {
  run "save-logs" {
    logfile = log.file
  }
}

# ...

job "save-logs" {
  option "logfile" {
    type = string
  }

  exec {
     command = "upload-to-jira"
     args = ["-f", opt.logfile]
  }
}

See the logcollection example for the full declaration of this command for reference.

Slack integration

Slack integration turns your Variant command into a Slack bot.

The bot is installed onto a specific Slack channel as a Slack app to listen any message to trigger a command.

Once you run:

SLACK_BOT_TOKEN=... SLACK_VERIFICATION_TOKEN=... variant start slackbot -n mycmd

in a directory that has a valid Variant command, it starts a RTM session to listen messages posted to the Slack channel.

When a message starting with:

/mycmd

is posted to the channel, the bot inteprets

/mycmd <CMD> <FLAGS>

as:

variant run <CMD> <FLAGS>

and runs it.

Auto-prompting via interactive messages

One of cool features of the bot is that when you missed to specify values for certain options, it will automatically start a interactive session to let you select and input missing values within Slack. You don't need to remember all the flags nor repeat lengthy commands anymore.

Writing Commands

Variant uses its own configuration language based on the HashiCorp configuration language 2.

It is designed to allow concise descriptions of your command. The Variant language is declarative, describing an intended goal of the command rather than the steps to compose your command.

In addition to everything available via the native HCL syntax, Variant language provides the following HCL blocks and functions to declare your command.

• Blocks
• Functions
• Examples

Blocks

• Job
• Parameter
• Option
• Run
  • Indirect Run
• Exec
• Config
• Test
• Steps
• Asserts

job

job "CMD SUBCMD {} is a job that can be run via run "CMD SUBCMD" {} or variant run CMD SUBCMD

Do only one thing in each "job"

Each job can contain any of the followings, but not two or more of them:

• assert
• exec
• run
• step

This restriction ensures that you can do only one thing in each job, which makes testing and reading the code easier.

That is, a job containing assert can be used as a custom assertion "function" and nothing else.

A job containing one or more steps can be used as a workflow composed of multiple jobs. Each step is restricted to call a single job. As each job is easily unit testable, this ensures that you can test the workflow without dealing with each job's implementation.

Options:

job has the following attributes:

• private: when set to true by writing private = true, the job is hidden from the command-line help.

parameter

parameter "NAME" {} is Nth positional argument to job that can be pased via run "the job" { NAME = "val1" } or variant run the job val1

option

option "NAME" {} is a named argument to job that can be passed via run "the job" { NAME = "val1" } or varuant run the job --NAME val1

config

config "NAME" {} is a layered configuration named NAME

Let's assume defaults.yaml has the following content:

api:
  endpoint: api.example.com
replicas: 2

And a Variant command that contains a job named thing that has a config block named conf:

job "thing" {
  option "env" {
    type = string
  }

  config "myconf" {
    source file {
      path = "defaults.yaml"
    }

    source job {
      name = "values"
      args = {
        env = opt.env
      }
    }

    source job {
      name = "value"
      args = {
        
      }
      key = "key1"
      format = "text"
    }
  }

  exec {
    command = "echo"
    args = list("apiendpoint=${conf.myconf.api.endpoint}, replicas=${conf.myconf.replicas}, env=${conf.myconf.env}, key1=${conf.myconf.key1}")
  }
}

job "values" {
  option "env" {
    type = string
  }

  exec {
    command = "echo"
    args = list("env: ${opt.env}")
  }
}

job "value" {
  exec {
    command = "echo"
    args = list("val1")
  }
}

The first source loads a file at defaults.yaml and merges the keys and values in it into conf.

Similarly, the second source loads keys and values from a job named values and merges them into conf.

The third sets the value for the key key1 where the value is the output of a job named value.

The configuration myconf can be referenced by ${conf.myconf.KEY} as shown in the exec block within the thing job above.

Let's run variant run thing --env prod and the output should be:

apiendpoint=api.example.com, replicas=2, env=prod, key1=val1

run

run runs a job with args. run is available within job and test.

job "a job" {
  run "another job" {
    param1 = "val1"
    opt1 = "val2"
  }
}

job "another job" {
  parameter "param1" {
    type = string
  }
  option "opt1" {
    type = string
  }

  exec {
    // ...
  }
}

Indirect Run

Sometimes you'd encounter a situation that you need indirection.

That is, the job name must be dynamically determined depending on a variable, parameter, or an option.

Variant2 provides an alternative run block syntax for that, which basically omits the NAME label in the run "NAME" { } you've previously seen:

To call the example job with the parameter foo set to "FOO" and the option bar set to "BAR", you write the run block like the below:

run {
  job = "example"

  with = {
    foo = "FOO"
    bar = "BAR"
  }
}

step

TODO

exec

An exec block executes the OS command.

Available attributes:

• cmd: The path to the executable binary/script
• args: The arguments to be passed to the command
• env: The environment variables given to the command
• dir: The working directory

Functions

• All the Terraform built-in functions
• Plus a few Variant-specific functions
  • jsonpath (definition, example)

Examples

To learn more, see examples for working examples covering all these blocks and functions.

Optionally read the following for overviews on each type of block and functions.

JSON Configuration Syntax

Variant has a JSON-based dialect of its DSL. It's based on the HCL's build-in feature to natively support JSON, so the conversion rules between HCL and JSON is very similar to famous HCL-based language like Terraform.

Generally speaking, you can use blocks, attributes and expressions with a little overhead. See the options-json example for more details.

Debugging Commands

Setting the environment variable VARIANT_TRACE to a non-empty value enables the trace logging.

It might be handy when you are curious how a third-party Variant command works, or when it seems like your command has an unknown bug that must be fixed.

Example:

$ VARIANT_TRACE=1 VARIANT_DIR=examples/issues/8-logging ./variant run example
go build -o variant ./pkg/cmd
foobar
TRACE   {"Type":"exec","Time":"2020-04-09T16:01:37.436145+09:00","Run":null,"Exec":{"Command":"echo","Args":["foobar"]}}exec={"args":["foobar"],"command":"echo"}

Writing Tests

Variant has its own testing framework composed of the test runner and the config syntax.

The test runner can be triggered by executing variant test. Upon run, the runner searches for test files whose names suffixed with _test.variant , parses it and run all the tests defined within it.

The test configuration syntax is as simple as declaring test "CMD1 SUBCMD1 {}, which is a set of unit tests for job "CMD1 SUBCMD1".

In each test, you can use case, run and assert blocks to define tests.

case "case1" { varname = value } defines a test case named case1 with the test data consists of a single variable called varname whose value is value. You can include one or more variables in a case.

run "jobname" { arg1 = value } specifies which job is run and which args is passed to the job run. This syntax is equivalent to run under jobs and steps.

Each assert { condition = expression } block adds a assertion that is run by the test framework. When the expression contained in the condition evaluated to false, the test fails.

Example:

test "options" {
  case "ok1" {
    exitstatus = 0
    err = ""
    out = trimspace(<<EOS
1 2 3 a b|c
EOS
    )
  }

  run "test" {
    int1 = 1
    ints1 = list(2,3)
    str1 = "a"
    strs1 = list("b","c")
  }

  assert "error" {
    condition = run.err == case.err
  }

  assert "out" {
    condition = (run.res.set && run.res.stdout == case.out) || !run.res.set
  }

  assert "exitstatus" {
    condition = run.res.exitstatus == case.exitstatus
  }
}

For more examples, please browse examples directories and look for any files ending with _test.variant.

Learning materials

The Variant DSL is very much based on HCL and CTY.

The following is a set of recommended reading for learning HCL and CTY.

hcl

• https://github.com/apparentlymart/terraform-clean-syntax

cty

• https://github.com/zclconf/go-cty