Feature: network use native system stores (#444)

By default, a bundle of SSL certificates is used, through
[certifi](https://pypi.org/project/certifi/). This PR allows QDT to use
the system certificates store.

In this PR:

- add
[truststore](https://truststore.readthedocs.io/en/latest/#using-truststore-with-requests)
as dependency to use new Python APIs (3.10) to deal with native system
certs stores
- add a new `QDT_SSL_USE_SYSTEM_STORES` environment variable to enable
this feature as an opt-in
- add related documentation

Related documentation:
https://guts.github.io/qgis-deployment-cli/guides/howto_behind_proxy.html#id1

cc @DeCiZoR and @Niarolf

docs/guides/howto_behind_proxy.md

@@ -42,30 +42,3 @@ At the shell session scope:
 > $env:QDT_PROXY_HTTP='http://user:password@proxyserver.intra:8765'
 > qdt -vvv
 ```
-
-----
-
-## Defining custom SSL client certificates
-
-Using a proxy for https connections typically requires the local machine to trust the proxy’s root certificate.
-
-> See [Requests official documentation](https://docs.python-requests.org/en/latest/user/advanced/#ca-certificates)
-
-### Using `REQUESTS_CA_BUNDLE` or `CURL_CA_BUNDLE`
-
-Point to a certificat bundle file path (*.pem).
-
-#### Example on Windows PowerShell
-
-Only for the QDT command scope:
-
-```powershell
-$env:REQUESTS_CA_BUNDLE="$env:USERPROFILE\cacerts.pem"; qdt -vvv
-```
-
-At the shell session scope:
-
-```powershell
-> $env:REQUESTS_CA_BUNDLE="$env:USERPROFILE\cacerts.pem"
-> qdt -vvv
-```

docs/guides/howto_use_custom_ssl_certs.md

@@ -0,0 +1,49 @@
+# Defining custom SSL client certificates
+
+Using a [proxy](./howto_behind_proxy.md) for https connections typically requires the local machine to trust the proxy’s root certificate. By default, a bundle of SSL certificates is used, through [certifi](https://pypi.org/project/certifi/) (using Mozilla curated list).
+
+> See [Requests official documentation](https://docs.python-requests.org/en/latest/user/advanced/#ca-certificates)
+
+Here comes how to customize which certificates bundle to use or how to require QDT to use the native system certificates store.
+
+## Using `REQUESTS_CA_BUNDLE` or `CURL_CA_BUNDLE`
+
+Point to a certificat bundle file path (*.pem).
+
+### Example on Windows PowerShell
+
+Only for the QDT command scope:
+
+```powershell
+$env:REQUESTS_CA_BUNDLE="$env:USERPROFILE\cacerts.pem"; qdt -vvv
+```
+
+At the shell session scope:
+
+```powershell
+> $env:REQUESTS_CA_BUNDLE="$env:USERPROFILE\cacerts.pem"
+> qdt -vvv
+```
+
+## Using native system certificates store
+
+If the `QDT_SSL_USE_SYSTEM_STORES` environment variable is set to `True`, HTTPS requests rely on the native system certificates store.
+
+:::{info}
+If enabled, this environment variable take precedence over `REQUESTS_CA_BUNDLE` or `CURL_CA_BUNDLE` which are ignored.
+:::
+
+### Example on Windows PowerShell
+
+Only for the QDT command scope:
+
+```powershell
+$env:QDT_SSL_USE_SYSTEM_STORES=true; qdt -vvv
+```
+
+At the shell session scope:
+
+```powershell
+> $env:QDT_SSL_USE_SYSTEM_STORES=true
+> qdt -vvv
+```

docs/index.md

@@ -60,6 +60,7 @@ caption: Guides
 maxdepth: 1
 ---
 guides/howto_behind_proxy
+guides/howto_use_custom_ssl_certs
 guides/howto_qgis_get_plugin_id
 guides/howto_schedule_deployment
 guides/howto_windows_sign_executable

docs/usage/settings.md

@@ -9,7 +9,7 @@ Some options and arguments can be set with environment variables.
 | Variable name                       | Corresponding CLI argument | Default value      |
 | :---------------------------------- | :------------------------: | :----------------: |
 | `QDT_LOGS_LEVEL`                    | `-v`, `--verbose` | `1` (= `logging.WARNING`). Must be an integer. |
-| `QDT_PROXY_HTTP`                    | `--proxy-http` | No proxy. |
+| `QDT_PROXY_HTTP`                    | `--proxy-http` to customize network proxy to use. See also [How to use behind a proxy](../guides/howto_behind_proxy.md). | No proxy. |
 | `QDT_SCENARIO_PATH`                 | `--scenario` in `deploy`   | `scenario.qdt.yml` |
 | `QDT_UPGRADE_CHECK_ONLY`            | `-c`, `--check-only` in `upgrade`   | `False` |
 | `QDT_UPGRADE_DISPLAY_RELEASE_NOTES` | `-n`, `--dont-show-release-notes` in `upgrade`   | `True` |
@@ -24,13 +24,15 @@ Some others parameters can be set using environment variables.
 | `QDT_LOCAL_WORK_DIR` | Local folder where QDT download remote resources (profiles, plugins, etc.) | `~/.cache/qgis-deployment-toolbelt/default/` |
 | `QDT_LOGS_DIR` | Folder where QDT writes the log files, which are automatically rotated. | `~/.cache/qgis-deployment-toolbelt/logs/` |
 | `QDT_QGIS_EXE_PATH` | Path to the QGIS executable to use. Used in shortcuts. | `/usr/bin/qgis` on Linux and MacOS, `%PROGRAMFILES%/QGIS 3.28/bin/qgis-ltr-bin.exe` on Windows. |
+| `QDT_SSL_USE_SYSTEM_STORES` | By default, a bundle of SSL certificates is used, through [certifi](https://pypi.org/project/certifi/). If this environment variable is set to True, QDT tries to uses the system certificates store. Based on [truststore](https://truststore.readthedocs.io/). See also [How to use custom SSL certificates](../guides/howto_use_custom_ssl_certs.md).  | `False` |
 
 ----
 
-## QGIS environment variables
+## 3rd party environment variables
 
-Some of the QGIS environment variable applies to QDT:
+Some of the 3rd party environment variable applies to QDT:
 
 | Variable name       | Description            |
 | :------------------ | :----------------------: |
+| `REQUESTS_CA_BUNDLE` | Set the path to the bundle of SSL certificates to use for HTTPS requests. See also [How to use custom SSL certificates](../guides/howto_use_custom_ssl_certs.md).  |
 | `QGIS_CUSTOM_CONFIG_PATH` | Used to customize the path to the folder where QGIS stores the user's profiles. See [upstream documentation](https://docs.qgis.org/3.34/en/docs/user_manual/introduction/qgis_configuration.html#profiles-path). |

qgis_deployment_toolbelt/utils/file_downloader.py

@@ -7,9 +7,11 @@
 
 # standard library
 import logging
+from os import getenv
 from pathlib import Path
 
 # 3rd party
+import truststore
 from requests import Session
 from requests.exceptions import ConnectionError, HTTPError
 from requests.utils import requote_uri
@@ -18,6 +20,7 @@
 from qgis_deployment_toolbelt.__about__ import __title_clean__, __version__
 from qgis_deployment_toolbelt.utils.formatters import convert_octets
 from qgis_deployment_toolbelt.utils.proxies import get_proxy_settings
+from qgis_deployment_toolbelt.utils.str2bool import str2bool
 
 # ############################################################################
 # ########## GLOBALS #############
@@ -26,6 +29,9 @@
 # logs
 logger = logging.getLogger(__name__)
 
+if str2bool(getenv("QDT_SSL_USE_SYSTEM_STORES", False)):
+    truststore.inject_into_ssl()
+    logger.debug("Option to use native system certificates stores is enabled.")
 
 # ############################################################################
 # ########## FUNCTIONS ###########

qgis_deployment_toolbelt/utils/journalizer.py

@@ -10,7 +10,7 @@
 import logging
 from getpass import getuser
 from logging.handlers import RotatingFileHandler
-from os import getenv
+from os import environ, getenv
 from pathlib import Path
 from platform import architecture, platform, uname
 from socket import gethostname
@@ -29,6 +29,7 @@
 from qgis_deployment_toolbelt.__about__ import __title__, __version__
 from qgis_deployment_toolbelt.constants import get_qdt_logs_folder
 from qgis_deployment_toolbelt.utils.proxies import get_proxy_settings
+from qgis_deployment_toolbelt.utils.str2bool import str2bool
 
 # ############################################################################
 # ########## GLOBALS #############
@@ -141,6 +142,21 @@ def headers():
         f"Certificate authority (CA) bundle to use: {getenv('REQUESTS_CA_BUNDLE', getenv('CURL_CA_BUNDLE'))}"
     )
 
+    if str2bool(getenv("QDT_SSL_USE_SYSTEM_STORES", False)):
+        logger.debug("Option to use native system certificates stores is enabled.")
+        if "REQUESTS_CA_BUNDLE" in environ:
+            environ.pop("REQUESTS_CA_BUNDLE")
+            logger.debug(
+                "Custom path to CA Bundle (REQUESTS_CA_BUNDLE) has been removed from "
+                "environment variables."
+            )
+        if "CURL_CA_BUNDLE" in environ:
+            environ.pop("CURL_CA_BUNDLE")
+            logger.debug(
+                "Custom path to CA Bundle (CURL_CA_BUNDLE) has been removed from "
+                "environment variables."
+            )
+
 
 def get_logger_filepath() -> Path | None:
     """Retrieve log filepath within logger handlers.

requirements/base.txt

@@ -6,3 +6,4 @@ packaging>=20,<24
 pyyaml>=5.4,<7
 pywin32==306 ; sys_platform == 'win32'
 requests>=2.31,<3
+truststore>=0.8,<1