Make working with SSL easier.

[karpetrosyan]

I believe the current SSL configurations are somewhat complicated, so we can consider simplifying things here.

As Tom mentioned in #947, one of the solutions is:

The SSL configuration using verify/cert/trust_env is somewhat overloaded. I can see that a neater approach would be a single ssl_context=... parameter.

To be honest, this approach does not appear to be user friendly enough.
Users must create an SSL context each time they want to simply disable SSL validation, which is not a fun thing to do, especially for people who don't know what an SSL context is.

Example:

import ssl
import httpx

ssl_context = ssl.create_default_context()
ssl_context.check_hostname = False
ssl_context.verify_mode = ssl.CERT_NONE

httpx.Client(ssl_context=ssl_context)

Instead, we can consider keeping the current approach but making some changes, such as removing the cert parameter entirely, and users who want to use client side certificates should consider creating their own ssl context and passing it through httpx.Client.

Instead of

import httpx

httpx.Client(cert="certfile_path")

do

import ssl
import httpx

ssl_context = ssl.create_default_context()
ssl_context.load_cert_chain(certfile="certfile_path")
# maybe also load ca bundle

httpx.Client(verify=ssl_context)

Also, because the user who wants to use other certificates appears to be able to configure the SSL context himself, we can deprecate passing the ca bundle path to verify.

The current approach is what the requests library does, but we could consider switching to an aiohttp-like API with only one ssl argument, which can be None, ssl context, or False.

None indicates that the user wishes to use the default configuration.
False indicates that the user simply wishes to disable SSL verification, whereas SSL Context indicates that the user provides its own configured SSL context and that the library should not perform any implicit configurations on top of this ssl context.

We can deprecate the verify argument and instead recommend that users use the ssl argument, and we can completely remove the verify argument with the next major version.

What are your thoughts? @encode/maintainers

[tomchristie]

Great, yup!

What I'd been planning here was reusing the existing verify/cert API, but pushing that configuration into an httpx.SSLContext(...) class. This would be a subclass of ssl.SSLContext() that provides the initial defaults based on the verify/cert config.

Adapting our existing documentation here's what I've got...

---

SSL certificates

When making a request over HTTPS, HTTPX needs to verify the identity of the requested host. To do this, it uses a bundle of SSL certificates (a.k.a. CA bundle) delivered by a trusted certificate authority (CA).

Enabling and disabling verification

By default httpx will verify HTTPS connections, and raise an error for invalid SSL cases...

>>> httpx.get("https://expired.badssl.com/")
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)

You can configure the verification using httpx.SSLContext().

>>> ssl_context = httpx.SSLContext()
>>> ssl_context
<httpx.SSLContext [verify=True]>
>>> httpx.get("https://www.example.com", ssl_context=ssl_context)
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)

For example, you can use this to disable verification completely and allow insecure requests...

>>> no_verify = httpx.SSLContext(verify=False)
>>> no_verify
<httpx.SSLContext [verify=False]>
>>> httpx.get("https://expired.badssl.com/", ssl_context=no_verify)
<Response [200 OK]>

Configuring client instances

If you're using a Client() instance, then you should pass any SSL settings when instantiating the client.

>>> ssl_context = httpx.SSLContext()
>>> client = httpx.Client(ssl_context=ssl_context)

The client.get(...) method and other request methods on a Client instance do not support changing the SSL settings on a per-request basis.

If you need different SSL settings in different cases you should use more that one client instance, with different settings on each. Each client will then be using an isolated connection pool with a specific fixed SSL configuration on all connections within that pool.

Changing the verification defaults

By default, HTTPX uses the CA bundle provided by Certifi.

The following all have the same behaviour...

Using the default SSL context.

>>> client = httpx.Client()
>>> client.get("https://www.example.com")
<Response [200 OK]>

Using the default SSL context, but specified explicitly.

>>> default = httpx.SSLContext()
>>> client = httpx.Client(ssl_context=default)
>>> client.get("https://www.example.com")
<Response [200 OK]>

Using the default SSL context, with verify=True specified explicitly.

>>> default = httpx.SSLContext(verify=True)
>>> client = httpx.Client(ssl_context=default)
>>> client.get("https://www.example.com")
<Response [200 OK]>

Using an SSL context, with certifi.where() explicitly specified.

>>> default = httpx.SSLContext(verify=certifi.where())
>>> client = httpx.Client(ssl_context=default)
>>> client.get("https://www.example.com")
<Response [200 OK]>

For some advanced situations may require you to use a different set of certificates, either by specifying a PEM file:

>>> custom_cafile = httpx.SSLContext(verify="path/to/certs.pem")
>>> client = httpx.Client(ssl_context=custom_cafile)
>>> client.get("https://www.example.com")
<Response [200 OK]>

Or by providing an certificate directory:

>>> custom_capath = httpx.SSLContext(verify="path/to/certs")
>>> client = httpx.Client(ssl_context=custom_capath)
>>> client.get("https://www.example.com")
<Response [200 OK]>

These usages are equivelent to using .load_verify_locations() with either cafile=... or capath=....

Client side certificates

You can also specify a local cert to use as a client-side certificate, either a path to an SSL certificate file...

>>> cert = "path/to/client.pem"
>>> ssl_context = httpx.SSLContext(cert=cert)
>>> httpx.get("https://example.org", ssl_context=ssl_context)
<Response [200 OK]>

Or two-tuple of (certificate file, key file)...

>>> cert = ("path/to/client.pem", "path/to/client.key")
>>> ssl_context = httpx.SSLContext(cert=cert)
>>> httpx.get("https://example.org", ssl_context=ssl_context)
<Response [200 OK]>

Or a three-tuple of (certificate file, key file, password)...

>>> cert = ("path/to/client.pem", "path/to/client.key", "password")
>>> ssl_context = httpx.SSLContext(cert=cert)
>>> httpx.get("https://example.org", ssl_context=ssl_context)
<Response [200 OK]>

These configurations are equivalent to using .load_cert_chain().

Using alternate SSL contexts

You can also use an alternate ssl.SSLContext instances.

For example, using the truststore package...

import ssl
import truststore
import httpx

ssl_context = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
client = httpx.Client(ssl_context=ssl_context)

Or working directly with Python's standard library...

import ssl
import httpx

ssl_context = ssl.create_default_context()
client = httpx.Client(ssl_context=ssl_context)

Working with SSL_CERT_FILE and SSL_CERT_DIR

Unlike requests, the httpx package does not automatically pull in the environment variables SSL_CERT_FILE or SSL_CERT_DIR. If you want to use these they need to be enabled explicitly.

For example...

# Use `SSL_CERT_FILE` or `SSL_CERT_DIR` if configured, otherwise use certifi.
verify = os.environ.get("SSL_CERT_FILE", os.environ.get("SSL_CERT_DIR", True))
ssl_context = httpx.SSLContext(verify=verify)

Working with SSLKEYLOGFILE

This environment variable is used for inspecing and debugging SSL.

Unlike requests or the standard library ssl.create_default_context the httpx package does not automatically configure an SSL context to use SSLKEYLOGFILE. If you want to use this it needs to be configured explicitly.

For example...

example.py:

import os
import httpx

def create_client():
    # Setup our SSL context
    ssl_context = httpx.SSLContext()
    keylog_filename = os.environ.get("SSLKEYLOGFILE")
    if keylog_filename:
        ssl_context.keylog_filename = keylog_filename

    # Create a client instance
    return httpx.Client(ssl_context=ssl_context)

client = create_client()
client.get("https://google.com")

We can now enable SSL key logging...

$ # Run the above example with SSLKEYLOGFILE debugging enabled.
$ SSLKEYLOGFILE=test.log python example.py
$ # Inspect the TLS secrets log file.
$ cat test.log
SERVER_HANDSHAKE_TRAFFIC_SECRET XXXX
EXPORTER_SECRET XXXX
SERVER_TRAFFIC_SECRET_0 XXXX
CLIENT_HANDSHAKE_TRAFFIC_SECRET XXXX
CLIENT_TRAFFIC_SECRET_0 XXXX
SERVER_HANDSHAKE_TRAFFIC_SECRET XXXX
EXPORTER_SECRET XXXX
SERVER_TRAFFIC_SECRET_0 XXXX
CLIENT_HANDSHAKE_TRAFFIC_SECRET XXXX
CLIENT_TRAFFIC_SECRET_0 XXXX

Making HTTPS requests to a local server

When making requests to local servers, such as a development server running on localhost, you will typically be using unencrypted HTTP connections.

If you do need to make HTTPS connections to a local server, for example to test an HTTPS-only service, you will need to create and use your own certificates. Here's one way to do it:

1. Use trustme to generate a pair of server key/cert files, and a client cert file.
2. Pass the server key/cert files when starting your local server. (This depends on the particular web server you're using. For example, Uvicorn provides the --ssl-keyfile and --ssl-certfile options.)
3. Tell HTTPX to use the certificates stored in client.pem:

>>> import httpx
>>> ssl_context = httpx.SSLContext(verify="/tmp/client.pem")
>>> r = httpx.get("https://localhost:8000", ssl_context=ssl_context)
>>> r
Response <200 OK>

[karpetrosyan]

Thank you for such a detailed explanation. I liked both approaches!

[karpetrosyan]

Do we need to have a discussion about which is better, ssl or ssl_context, or is the answer obvious?

[tomchristie]

Don't know. ssl=... slightly nudges users towards accidentally shadowing the stdlib name. So... maybe the longer form is better? Tho???

[karpetrosyan]

Then I guess httpx.SSLContext should be just a function, or httpx.create_ssl_context, which I believe would be much cleaner than subclassing ssl.SSLContext, am I correct?

[tomchristie]

I think it'll be neatest as a subclass of ssl.SSLContext with configuration on __init__ and with a nice __repr__ indicating some of the state.

I've updated the proposed docs with a bit of improving the structure, and with notes on working with the various environment variables that we currently support. (You'll notice that I'm proposing we drop trust_env=... here, and be explicit instead.)

[agronholm]

Just for the record, AnyIO treats tls=False to connect_tcp() as "don't do a TLS handshake". It may not be relevant here, as the context is somewhat different.

[tomchristie]

Ah that's a good point actually @agronholm. Yup clearer to have an explicit SSLContext(verify=False), rather than an ...=False shortcut.