Q: What's the correct way to setup statically managed domains + dynamically added domains?

[zllovesuki]

Use case: I have a project that allows users to create reverse tunnel over QUIC, and their tunnels will be available on tunnel.example.com, such as string-string-string.tunnel.example.com. However, I also want to allow user to point their own domain to the service.

Currently there's ManageAsync on tunnel.example.com and *.tunnel.example.com, with only DNS solver available. Assuming that the "custom hostname" is web.mydomain.com, and its TXT record is managed by the service, how would you dynamically add that to a running certmagic instance?

[mholt]

You'd want to use on-demand TLS: https://pkg.go.dev/github.com/caddyserver/certmagic#Config.OnDemand

Probably create two certmagic configs, one that manages the hostnames you control, and one for the on-demand config, and have both configs share the same cache.

[zllovesuki]

Does that mean I need to return the correct Config via the cache's GetConfigForCert?

Follow-ups:

1. Can the two configs share the same storage?
2. Can the two configs share the same issuer (DNS)?

[mholt]

Does that mean I need to return the correct Config via the cache's GetConfigForCert?

Yep! Because during a certificate's lifetime, how it is managed may change, so that callback is invoked at the time of renewal to ensure the most current, correct config is used.

Can the two configs share the same storage?

Yes!

Can the two configs share the same issuer (DNS)?

Also yes!

[zllovesuki]

Do I need to mux the GetCertificate as well when using with *tls.Config?

[mholt]

Sorry I missed this reply until now.

What do you mean by mux it? Can you explain your question a little more?

[zllovesuki]

Sorry, I meant if I need to call the correct GetCertificate from different config based on the handshake

[zllovesuki]

OK, now it is dynamically selecting between the two configs. However, there's an issue:

2023-04-15T00:54:58.482-0700 DEBUG acme_client acme/http.go:270 http request {"component": "acme_dynamic_issuer", "method": "POST", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/6126350974", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["98214744"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["821"],"Content-Type":["application/json"],"Date":["Sat, 15 Apr 2023 07:54:58 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["BEB95Xq7ssSvBBwCUvzNemBK_K9Ea_iTkat9X1V24tHoIOI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2023-04-15T00:54:58.482-0700 ERROR obtain certmagic@v0.17.2/config.go:567 could not get certificate from issuer {"component": "acme_dynamic", "identifier": "dev.con.nect.sh", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[dev.con.nect.sh] solving challenges: presenting for challenge: presenting with embedded solver: could not start listener for challenge server at :443: listen tcp :443: bind: permission denied (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/98214744/8275522584) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}

It seems like if I'm using the built-in solvers (http or tls), certmagic will attempt to start listeners locally. However I already have port 80 and 443 listening in the application that can handle them. Is there a way to work around that? Or do I have to use dns solver only?

[francislavoie]

How are you setting it up in code? You should just pull the TLS Config from certmagic and use it with your own listener; don't call certmagic.HTTPS etc. See https://github.com/caddyserver/certmagic#tls-alpn-challenge

[mholt]

It seems like if I'm using the built-in solvers (http or tls), certmagic will attempt to start listeners locally. However I already have port 80 and 443 listening in the application that can handle them. Is there a way to work around that? Or do I have to use dns solver only?

@zllovesuki CertMagic will only try to bind the port if the socket isn't already in use, but it's tricky in this case since it looks like your program doesn't have permission to bind port 80/443. When it tries binding the port to see if it's in use (and to acquire the socket if it's not), it will defer to whatever is already running if it gets an error like "already in use" -- but in your case it's getting "permission denied" which is a different error.

Try giving your program permission to bind to ports 80/443 and it should work.

[zllovesuki]

There's a workaround to:

1. start a dummy listener on port X;
2. specify the tls alt port to X;
3. I think certmagic will see that bind failed and do a fall through on the handling, because the application already handles the TLS already.

However, in my use case I still need the DNS approach, as that's also being used to verify the ownership in the application.

[mholt]

You should still be able to use the DNS challenge even with that workaround -- does changing the permissions work at all too?