Security of root CA keys in clusters deployed by the SCS reference implementation

[horazont]

kubeadm, by default, writes the Root CAs of the k8s cluster in plaintext on the control plane nodes. This affects three CAs:

• The k8s main CA, which allows to impersonate all entities in a cluster, incl. nodes and the admin user.
• The k8s front proxy CA, which allows to impersonate the k8s API server when talking to API integrations.
• The etcd CA, which allows to impersonate etcd nodes and clients.

(If calico is used, it may also have its own CA which allows to impersonate calico components among each other.)

From this approach, the following issues follow:

• The Root CA key for each of those is unprotected (only posix permissions) on every control plane node.
• There is no story for rolling over the Root CA key after its default lifetime of 10 years.

The second issue may be handwaved away with "no k8s cluster is going to live for 10 years", even though I disagree on that stance.

We should figure out which options we have to make our clusters more secure. In particular, we should avoid having the Root CA on the k8s node. Abstractly, this breaks auto-renewal of certificates and makes the join process more complicated: the k8s API server needs to be able to sign certificates in order to let a node join.

As food for thought, in yaook/k8s, C&H is experimenting with using HashiCorp Vault to keep the CA keys secure and using Vault policies and ACLs in order to restrict the certificates individual nodes may obtain to the absolute minimum. This already works to the point that fully functional cluster can be obtained, but it's not in production yet.

We are not using the openstack C-API provider directly, but implement the C-API ourselves using our own ansible-based LCM, so the changes I propose there are probably not 1:1 applicable to the openstack provider.

In addition, this creates a hard dependency on HashiCorp Vault, which may or may not be acceptable; the good thing about Vault is that it supports rather high-level operations on PKI objects (e.g. "issue a certificate"), which is better than what most (if not all) other secret storage engines can do.

[garloff]

Need more discussion whether we can have a stronger dependency on Vault.
Unsealing strategy needs to be defined as well. (HSM unsealing only in Vault Enterprise.)
(KSM services in clouds are a separated discussion.)
-> Discussion in overflow slot 2022-08-22 15:05 -- 16:25.

passwordstore.org / git-crypt / sops may be insufficient (and don't compete with vault)

[joshmue]

Regarding your HSM notes, it seems to me as if the use case outlined by the alternatives is very different from the Vault use case.

To get on the same page here:
Would you roughly agree to this use case definition: "We want some sort of PKI which issues certificates based on identity of the requester"?

Regarding cert-manager: Actually, you can use just the CertificateRequest resource which just does the signing and does AFAIK not deal with the private part of a certificate at all and allows to implement certificate approval. It can be used as a kind-of imperative API in that regard. Did not touch the approval part yet, but seems simple enough.
The chicken-egg problem would need to be addressed of course. There would have to be some management cluster which is not provisioned by the same mechanisms as the workload clusters.

Regarding the restriction of endpoints to sign certificates for: I guess that is all similar for cert-manager, Vault, step-ca and other tooling.
In all cases, some custom logic will create some kind of "role" which can be used by some kind of "identity" to let issue certain certificates.
It seems that, for step-ca OSS, such policies can only be applied at CA level, making the process more difficult, indeed.

---

Thanks a lot for the detailed insights! IMHO looks very thought out with nicely tight access controls.

Have one unified credential which identifies the node and which grants it access to all the endpoints it needs to sustain itself and to obtain further credentials which it can use to authenticate itself.

Even unrelated to the certificate issuance use case, I think that this is a goal worth pursuing (apart from "federating" K8s apiservers, which is another important topic).

Most big cloud providers offer assigning an identity to workloads like VM's. So if some VM needs to access some service requiring authentication, it can just use its identity which got corresponding permissions assigned in the central IAM system.

AFAIK, this is not really something that is handled by the OpenStack ecosystem itself. As far as I understand, you effectively implemented such kind of system: Assigning identities with scoped permissions (in this case for getting certificates from Vault) to machines.

One other simple use case for example: Grant workload K8s nodes permissions to pull from a to-be-implemented managed container registry.

[horazont]

Would you roughly agree to this use case definition: "We want some sort of PKI which issues certificates based on identity of the requester"?

... and which can handle at least a few dozen CAs. A single k8s cluster alone will need three to four CAs.

And also mind that "identity" is here not equivalent to the subject of the certificate to be issued. In particular, a control plane node may need certificates for system:nodes:$nodename, but also for the API server IP and hostname, which is also needed by the other control plane nodes. So there is no 1:1 mapping between certificate subjects and identities.

Regarding the restriction of endpoints to sign certificates for: I guess that is all similar for cert-manager, Vault, step-ca and other tooling.
In all cases, some custom logic will create some kind of "role" which can be used by some kind of "identity" to let issue certain certificates.

The difference from my perspective here is that Vault is made for proper, fine grained access control, with a DSL which is somewhat easy to review and audit.

cert-manager is to some extent bound by what k8s can deliver. For step-ca it still looks as if one would have to implement that access control system oneself if one wanted more fine-grained control than on the CA level. I think the same holds for cert-manager, if I'm reading the documentation correctly; you'd have to write an approval controller which somehow judges certificate requests based on the entity which requested it (which is, I think, not actually available in the CerificateRequest?).

And k8s RBACs are very non-obviously tricky. For instance, the pod create permission in a namespace is sufficient to escalate to the privileges of any service account in that namespace.

But you are right that CertificateRequest at least avoids storing the private key in k8s secrets, I wasn't aware of that option.

---

AFAIK, this is not really something that is handled by the OpenStack ecosystem itself. As far as I understand, you effectively implemented such kind of system: Assigning identities with scoped permissions (in this case for getting certificates from Vault) to machines.

Kind of. And thanks to Vaults diverse secrets engines, this trust can easily transferred to (some) other systems. Funnily enough, feeding that credential via instance user-data is on the todo list :-).

One other simple use case for example: Grant workload K8s nodes permissions to pull from a to-be-implemented managed container registry.

This could be implemented easily with client-cert authentication + a CA in vault + short-lived certificates issued on demand (or continuously renewed).

[joshmue]

A single k8s cluster alone will need three to four CAs.

I guess that's not fundamentally different for Vault, if deployed highly available and using transport encryption.
It will be less complex, as fewer components are involved, but I doubt that the difference in maintenance is huge.

HA in Vault requires a storage backend supporting HA. The swift backend (AFAIK an OpenStack core service) does not appear to be HA capable, so most probably one would use the "integrated (Raft) storage".
As far as I understand, that would require distributing some unmanaged internal certificates out-of-band to the Vault nodes.

For step-ca it still looks as if one would have to implement that access control system oneself if one wanted more fine-grained control than on the CA level.

That is how I understood the docs, too.

I think the same holds for cert-manager, if I'm reading the documentation correctly; you'd have to write an approval controller which somehow judges certificate requests based on the entity which requested it (which is, I think, not actually available in the CerificateRequest?).

According to docs, it is. I guess that the correct identity is put in via the cert-manager mutating admission webhook, but did not dive in that deep, yet.

And k8s RBACs are very non-obviously tricky. For instance, the pod create permission in a namespace is sufficient to escalate to the privileges of any service account in that namespace.

That's right. Fortunately though, permissions can be extremely narrow, too.

---

Ultimately, I think that Vault is the most intuitive choice for something like this, but not the only one that would fit such use case.

Coming back to the initial HSM consideration (for unsealing, signing or anything that could have benefits), I guess there are at least five different ways to go forward:

1. Using Vault; SCS does not plan to use HSM's
2. Using Vault; If a provider wants to use HSM's, SCS allows installing Vault Enterprise, but does not support it
3. Using Vault; If a provider wants to use HSM's, SCS supports installing Vault Enterprise
4. Using cert-manager (or something else); Put HSM usage in backlog
5. Using cert-manager (or something else); Official support for PKCS#11 HSM's by SCS

---

Anyhow, whatever the tooling, something like

Funnily enough, feeding that credential via instance user-data is on the todo list :-).

(plus the automatic rotation of certs etc.) sounds like a great thing to me.

[horazont]

A single k8s cluster alone will need three to four CAs.

I guess that's not fundamentally different for Vault, if deployed highly available and using transport encryption.
It will be less complex, as fewer components are involved, but I doubt that the difference in maintenance is huge.

No, I meant CAs. With k8s, you have the API frontend CA, the etcd CA and the API server proxy CA, all of which should be distinct. Calico needs its own internal CA, too.

HA in Vault requires a storage backend supporting HA. The swift backend (AFAIK an OpenStack core service) does not appear to be HA capable, so most probably one would use the "integrated (Raft) storage".
As far as I understand, that would require distributing some unmanaged internal certificates out-of-band to the Vault nodes.

Raft does not require distributing an unmanaged CA. Once the join succeeds (via a join challenge), Vault takes care of distributing the internal cluster TLS credentials automatically, according to the documentation.

I think the same holds for cert-manager, if I'm reading the documentation correctly; you'd have to write an approval controller which somehow judges certificate requests based on the entity which requested it (which is, I think, not actually available in the CerificateRequest?).

According to docs, it is. I guess that the correct identity is put in via the cert-manager mutating admission webhook, but did not dive in that deep, yet.

Ah, I did not see that when I skimmed the docs the other day. So that information is available, but it is limited to the identities in the k8s cluster. That means this requires an identity management tool outside of k8s (or even more carefulness with policies w.r.t. service account tokens), which is then to be trusted with the power to allow entities to sign certificates and which needs to know about all machines managed by that cluster.

Ultimately, what I do not like about the cert-manager approach is that it is a lot of moving parts which are involved in handling the trust and the secrets:

• You have the k8s API server to provide the final authorization
• Governed by RBAC policies which are spread throughout the namespace and cluster-wide rules
• Then there's whatever provides identity information to the k8s API server, for lack of internal user management; this will need to manage all machine accounts (so the central keycloak is probably not a good choice?). The trust between these two probably needs another CA, or at least a shared secret (JWT?).
• You have cert-manager resources which are protected by a webhook from modification; if that webhook is misconfigured, the configuration breaks (breaking version change for instance) in a way which makes it fail in a way which puts the webhook out of the loop, the user information and other stuff in the certificate flow can easily be spoofed by anyone with privileges to patch or create certificaterequests.
• Anything which can spawn pods can probably escalate far into the security perimeter of cert-manager a lot unless there's additional protection e.g. through (deprecated) PodSecurityPolicies or their successor or stuff like kyverno.

While I'm sure you can get that secure, and you can monitor all of this to prevent the worst, I don't feel like trusting it with cluster CA issuance...

As much as I like the simplicity of cert-manager for highly dynamic workloads (we use it extensively in Yaook/operator, e.g. for DB replication CAs, though I've wondered about replacing it for that use case too), I'm not so sure this is a great application.

I also think that these points weigh more strongly than the HSM support: HSM support is worthless if the API to issue certificates is open to too many entities.

[joshmue]

Raft does not require distributing an unmanaged CA. Once the join succeeds (via a join challenge), Vault takes care of distributing the internal cluster TLS credentials automatically, according to the documentation.

Did not know about the challenge bits (and overlooked them in the docs the first time)! Sounds like the smart thing to do.

---

As much as I like the simplicity of cert-manager for highly dynamic workloads (we use it extensively in Yaook/operator, e.g. for DB replication CAs, though I've wondered about replacing it for that use case too), I'm not so sure this is a great application.

I am not sure about that, either.

HSM support is worthless if the API to issue certificates is open to too many entities.

I also can 100% agree to that.
I just do not think that "is open to too many entities" would necessarily apply to a cert-manager based solution, even tough the system would be more complex, of course.

In the end, even one simple self-written server doing nothing more than starting an HTTPS server...

• authenticating requests bearing IdP tokens, which somehow are provisioned to machines
  • akin to authenticating via Vault tokens, which somehow are provisioned to machines
• authorizing requests
  • akin to using Vault policies
  • akin to cert-manager approvals
• signing certificate requests using a CA which resides either in memory (and encrypted on disk) or in an HSM (possibly using libraries wrapping PKCS#11 C code or openssl CLI, which might be considered risky)
  • akin to using Vault's PKI secrets engine

...would fulfill most needs.
Of course, the route of storing the CA on disk (encrypted with a single key) would most likely be inferior to Vault's shamir seal.
In turn, Vault+shamir might be considered inferior to a simple HSM backed signing server.
In turn, this might be considered inferior to using Enterprise Vault+HSM.

Not having any strong opinion about this order, of course. I just think that there are plenty of options, each having their own advantages and caveats, making the optimal trade off hard to find.

---

Then there's whatever provides identity information to the k8s API server, for lack of internal user management; this will need to manage all machine accounts (so the central keycloak is probably not a good choice?). The trust between these two probably needs another CA, or at least a shared secret (JWT?).

Wether or not machines should have "first-class identities", managed by (or federated to) the central IdP is indeed a very interesting question. IMHO, they should.

Authentication towards the issuing PKI server issuing the cluster certs would be one use case of such identity, of course.