Add support for updated Hashicorp HCP CLI to continue accessing Vault Secrets

[arrrgi]

Is your feature request related to a problem? Please describe.

The vlt CLI tool for interacting with HCP Vault Secrets has ended support as of September 2024. Whilst the CLI tool still functions, Hashicorp have advised users to migrate to HCP CLI, with this new package generally available for Linux, MacOS and Windows from https://developer.hashicorp.com/hcp/docs/vault-secrets/get-started/install-hcp-cli

The vlt CLI tool should be considered as deprecated now.

Describe the solution you'd like

Please add support for the HCP CLI, preferably using the existing hcpVaultSecrets variables so that users can migrate to the new CLI with less refactoring of their existing configs.

Describe alternatives you've considered

vlt continues to function but no further updates, especially security critical updates will be provided.

Additional context

[arrrgi]

Some additional information about the new HCP CLI and interacting with the Hashicorp Vault Secrets service:

• If I have created a project, for example, my project is called dotfiles, and
• if I have created a named application in my project, for example, my application is called chezmoi, and
• if I have created a Service Principal in my project, for example, an SP called chezmoi-viewer with the view role, and
• if I have created a credential set (HCP_CLIENT_ID and HCP_CLIENT_SECRET), and
• if I have created a secret called superSecret,

then it is sufficient to run:

export HCP_CLIENT_ID=xxxxxxxxx HCP_CLIENT_SECRET=xxxxxxxxx
hcp vault-secrets secrets list --app chezmoi
hcp vault-secrets secrets open superSecret --app chezmoi

whereas previously with the vlt CLI, the command required:

export HCP_CLIENT_ID=xxxxxxxxx HCP_CLIENT_SECRET=xxxxxxxxx
vlt secrets list --organization <organization UUID> --project <project UUID> --app-name chezmoi
vlt secrets get --organization <organization UUID> --project <project UUID> --app-name chezmoi superSecret

[halostatue]

Based on what you have written, the ability to skip --organization and --project would be due to the "service principal" and the associated HCP_CLIENT_ID/HCP_CLIENT_SECRET values.

I don't use Hashicorp Cloud products regularly, and while I did some testing of this the first time around, it was after the initial implementation was completed.

We can make this change (it would be contingent on the presence of hcp in the execution path as opposed to vlt in the execution path, which is not too different to how the 1Password CLI upgrade from v1 to v2 was achieved), but we’d need to know whether the command-line parameters for hcp allow the organization and project to be passed directly the way that vlt did. I’m not sure whether the HCP employee who interacted on #3067 is still there, but getting some feedback on this to help guide our development would be nice.

[arrrgi]

Could you explain please why organizationId and projectId are still required to implement this?

Perhaps these become optional instead of mandatory values so that users still with vlt can provide them in their config, and users of the new hcp CLI need only provide HCP_CLIENT_ID, HCP_CLIENT_SECRET as env config values, with the only other mandatory config being the hcpVaultSecrets/applicationName key value being set.

[arrrgi]

And if this is any further help, the hcp command takes a global --format=json | table | pretty (default) to format the output.

Further, there is no longer a .vlt.json file to store cached creds with hcp, these are now captured under a small number of files inside ~/.config/hcp, namely:

• .config/hcp/active_profile.hcl -> containing the name of the profile, defaults to name="default"
• .config/hcp/creds-cache.json -> containing the service principal ID and access token
• .config/profiles/default.hcl -> containing profile name, organization ID and project ID

These are all populated automatically in the condition that you have the Service Principal environment vars set and provide the --app-name xxx switch

[halostatue]

I can't answer this, because I’ve used Hashicorp Vault Secrets once, eighteen months ago, when this was first developed for chezmoi. However, chezmoi passes the provided items (secret name, app name, project id, and org id) as command-line parameters. If the parameters are substantially different, we need to create new template functions because the old functions will no longer work. If the expected items will also work with the new command line without breaking, then we can bodge the old functions to work the same.

Chezmoi does not explicitly disable configuration lookup for vlt, but neither does it parse or use cached configuration. This will be the same for the new hcp command, regardless.

[arrrgi]

Yes, I would think substantially different as demonstrated in the examples of vlt vs hcp vault-secrets above.

[halostatue]

TL;DR

The hcp command is an absolute mess and we will not be able to use the same template functions, because the organization id cannot be used at all; the project id can only be used to override to a different project in the same organization (and that's if you’re using an account instead of a "service principal", which is able to be substantially more restricted).

It cannot accept an organization id on the command-line and requires that the credentials be for a service entity or that you set up organization and project prior to use and specify a profile … I think. It's substantially more complex and is feels built for Hashicorp's overpaying enterprise customers, not for home lab use. The DX is atrocious and I can't really see how chezmoi is going to be able to properly work with this because it is both less programmatically usable and there are three distinct type of secrets (two of which cannot be tested with anything less than a "plus" account).

Implementation Notes

This should help someone who actually wants to implement this do so. I have no interest in implementing this after having spent the last couple of hours looking into it (sorry @arrrgi; it's that bad).

The Bad News

It's pretty much all bad news and the documentation is uncharacteristically atrocious for Hashicorp (I’m used to Terraform documentation and found the vlt documentation to be much clearer).

• There needs to be a new template function. There is zero compatibility between vlt and hcp, and given that there is no --format plain to just get the secret, there is only value in a template function that returns the --format json output as neither the --format pretty nor --format table is meaningfully parseable.
• Required parameters are secret_name and app_name. In my case, hcp vault-secrets secrets open test_secret --app chezmoi --format json. As with the current template, we could store a default app_name to make it optional, but I do not recommend it because it is not how hcp itself works.
• There are two main ways of authenticating the hcp command-line.
  • oauth-cli: All of my tests were done with this, meaning it had to route via the web once. I could not meaningfully connect to secrets until I did hcp profile init and it auto-discovered my single org and project and saved them.
  • service principal: You create a service principal "account"/"entity" that has its own HCP_CLIENT_ID and HCP_CLIENT_SECRET and set that prior to running hcp.
• Optional parameters that we could possibly care about are:
  • --project: A way of providing an alternate project ID for the used profile. I think. I only have the one project.
  • --profile: A way of specifying an entirely different organization and project combination by name. I think. I only have the one organization.
  • --profile appears to only matter with oauth-cli authentication.
  • --project may or may not work with a service principal; I could not see whether such a principal could be locked to a single project.

The returned data format is dynamic based on the underlying type of the secret, and there are no examples anywhere for what they might look like. The Swagger API definition does not help (the raw JSON doesn't match what's on the website, and what's on the docs website looks closer to the only output that I can get).

Here's an example:

{
  "created_at": "2023-06-28T01:55:18.686Z",
  "created_by_id": "<uuid>",
  "latest_version": 2,
  "name": "test_secret",
  "static_version": {
    "created_at": "2023-06-28T01:56:11.053Z",
    "created_by_id": "<uuid>",
    "value": "real-test-value",
    "version": 2
  },
  "type": "kv"
}

The value, however, appears to be polymorphic (see static_version above) with the following shapes seeming at least theoretically possible. Where the shape says object, I have hit a dead end on figuring out exactly what that would be. I have marked sync_status, static_version, rotating_version, and dynamic_version as optional, but I suspect that exactly one of static_version, rotating_version, or dynamic_version will be presented.

name: string # The name of the secret
type: string
latest_version: integer
created_at: string
created_by_id: string
sync_status: # optional?
  status: string
  updated_at: date-time
  last_error_code: string
static_version?:
  version: integer
  value: string
  created_at: string
  created_by_id: string
rotating_version: # optional
  version: integer
  values: object
  created_by_id: string
  created_at: string
  expires_at: string
  revoked_at: string
  keys: string[]
dynamic_version: # optional
  values: object
  created_at: string
  expires_at: string
  ttl: string

Because of the complexity of setup and the highly dynamic response possible with this, I don't think that this is a particularly good development investment and would not be recommending it to any developer that I know even if chezmoi did support it. (I feel the same way about the 1Password SDK because it only works with service principal equivalents.)

That said, if we do support it, we should put constraints:

• It only has a function that returns the JSON output. Determining whether the secret is static_version, rotating_version, or dynamic_version is the caller's responsibility. Any examples we provide can only use static_version unless we can get output samples of rotating_version and dynamic_version.
• The only parameters accepted are app_name and secret_name in that order, and they're both required: {{ hcpVaultSecretNew "app_name" "secret_name" }}
• The above should work with both service principal or oauth logins.
• There is no configuration as there is with the current hcp.

Regardless of anything else, the existing CLI has a limited shelf life. They are no longer updating it, and it is not clear if or when it will stop working. We should deprecate the existing template functions as they will fail at some point in the future.

[arrrgi]

Thanks @halostatue , great write up on what you found.

My own use case is to store static credentials/SSH keys, etc, and I do so with a Service Principal so that I don't need to have HCP CLI manually installed first and authenticated via OAuth before I init/apply my dotfiles. This was one of the reasons the hooks functionality was created so that my dotfiles could just be applied without further intervention than supplying HCP_CLIENT_ID, HCP_CLIENT_SECRET, Project and Org ID's.

See arrrgi/dotfiles for my current implementation using Service Principal and pre-install hook with vlt

HCP CLI simplifies this in invocation when using a Service Principal, and I would hope that this is the predominant use case for Chezmoi users consuming the existing hcpVaultSecret function as it removes the need for OAuth and browser combination --> hello Devcontainers/Codespaces users!!

Opinion: I don't imagine that rotating and dynamics secrets would be a common use case for Chezmoi users, that said, Hashicorp does provide free credit to test said secret types output if that is to be explored.

[halostatue]

In the short term, you can make this work by ensuring that you have HCP_CLIENT_ID and HCP_CLIENT_SECRET exported prior to running chezmoi and then use {{ output "hcp" "vs" "s" "open" "--format" "json" "--app" $app_name $secret_name | fromJson }}.

In my opinion, there is no meaningful way to use the existing functions.

• hcpVaultSecret relies on --format plain, which does not exist. The default is --format pretty, which requires parsing the output for the line ^Value:\s+(.+)$, more or less (and it is unclear whether it could be affected by localization). --format table puts the value in the fifth column, but as I only have the one secret, it is unclear whether the table is fixed width or responsive. That leaves --format json which has three fields where only one of them will be present, and only one of those has a value field, the others have an undocumented values object field.
• hcpVaultSecretJson could theoretically be used, but the JSON output is very different than what current users of the function would expect, and we would have to know that two of the arguments provided are effectively illegal when using hcp instead of vlt.

Coming up with a new name will be a pain, but the best choice is probably hcpVaultSecretV2 (even though hashicorp hasn't called it v2, the interface change is what I would call a breaking change) and it only supports --format json and leaves it to the caller to choose which of .static_version.value, .rotating_version.versions.blah, and .dynamic_version.versions.blah should be used.

I appreciate what you’re using this for, but Hashicorp seems to want its paying customers to use this through other integration mechanisms and the hcp CLI has apparently been provided as an afterthought. The retrieval secrets example references a command that does not exist (hcp vault-secrets secrets run) and it does so in a way that looks like it wouldn't work anyway (no --app specified).

---

There are other tools which provide 'service principal' approaches (AWS Secrets Manager, AWS SSM ParameterStore, 1Password) which could probably be used more reliably than what Hashicorp is providing here (and Chezmoi supports 1Password's approach somewhat natively with the SDK function), but as I don't use codespaces, I honestly don't have a reason to try them out with chezmoi.

I'm aware of Hashicorp's free credit, but there is also the matter of the time it would take to develop this feature. I am happy to support chezmoi users with my free time, but I’m less inclined to support Hashicorp after the licensing debacle last year. Had Hashicorp bothered to make the command-lines even remotely compatible, I would not have minded modifying chezmoi's approach. As it is, it involves a time investment that I have already overdrawn.

I’m not sure of anyone else's time commitments, but I don't see a reason that chezmoi shouldn't support this, but I won't be able to really even provide testing support with this change. @twpayne other contributors may feel differently.

[arrrgi]

{{ output "hcp" "vs" "s" "open" "--format" "json" "--app" $app_name $secret_name | fromJson }}

Let's say that I do use this for now, how would I access the value property? Like this?

[GDrive]
type = drive
client_secret = {{ (output "hcp" "vs" "s" "open" "--format" "json" "--app" $app_name $secret_name | fromJson).static_version.value }}

Alternatively, I would happily consider switching to the onePasswordSDK template functions if this is a built-in that doesn't require installation of 1Password SDK itself everywhere I need my dotfiles.

[twpayne]

Let's say that I do use this for now, how would I access the value property? Like this?

[GDrive]
type = drive
client_secret = {{ (output "hcp" "vs" "s" "open" "--format" "json" "--app" $app_name $secret_name | fromJson).static_version.value }}

Yes, exactly.

Alternatively, I would happily consider switching to the onePasswordSDK template functions if this is a built-in that doesn't require installation of 1Password SDK itself everywhere I need my dotfiles.

The 1Password SDK functions were added at the request of a Project Manager at 1Password, but I don't think anyone actually uses any of them yet (at least, searches for onepasswordSDKItemsGet and onepasswordSDKSecretsResolve on GitHub don't reveal any public dotfile repos using them).

If you want cross-platform secrets with no external dependencies, consider using age. chezmoi links age support as a library.

Medium-term, I'd like to add support for the new hcp command to chezmoi, but as I'm also not a Hashicorp user myself, so this may take some time.

[arrrgi]

The 1Password SDK functions were added at the request of a Project Manager at 1Password, but I don't think anyone actually uses any of them yet (at least, searches for onepasswordSDKItemsGet and onepasswordSDKSecretsResolve on GitHub don't reveal any public dotfile repos using them)

I have just tested and already received an error message.

chezmoi: .config/git/user: template: dot_config/exact_git/user.tmpl:5:18: executing "dot_config/exact_git/user.tmpl" at <onepasswordSDKSecretsResolve "op://Secrets/Personal SSH Signing Key/public key">: error calling onepasswordSDKSecretsResolve: error resolving secret reference: the specified field cannot be found within the item

Doing some digging, it seems the SDK is still pretty young and not all types of records can be retrieved yet with a number of record/field types not yet supported. Off-topic yes, but worthy of reporting given your mention that nobody else seems to be using this functionality yet.

Thanks for considering adding support, it's inelegant as @halostatue points out, but much simpler to setup than AWS/Azure for those of us with simpler needs.