Add retry logic to 'Download Miniforge' step of win pipeline

[maresb]

I'd like to solve this download failure and those like it by retrying.

Checklist

• Added a news entry

[maresb]

Since the logs get deleted after some time, here are the full logs from the link at the top:

2022-06-25T08:30:56.8153788Z ##[section]Starting: Download Miniforge
2022-06-25T08:30:56.8265003Z ==============================================================================
2022-06-25T08:30:56.8265569Z Task         : Python script
2022-06-25T08:30:56.8265883Z Description  : Run a Python file or inline script
2022-06-25T08:30:56.8266181Z Version      : 0.200.0
2022-06-25T08:30:56.8266449Z Author       : Microsoft Corporation
2022-06-25T08:30:56.8266839Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/python-script
2022-06-25T08:30:56.8267265Z ==============================================================================
2022-06-25T08:31:00.3052264Z [command]C:\hostedtoolcache\windows\Python\3.7.9\x64\python.exe D:\a\_temp\4f3a0980-d178-4d00-93eb-2671f8d307aa.py
2022-06-25T08:31:15.6520945Z Traceback (most recent call last):
2022-06-25T08:31:15.6521726Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\urllib\request.py", line 1350, in do_open
2022-06-25T08:31:15.6522305Z     encode_chunked=req.has_header('Transfer-encoding'))
2022-06-25T08:31:15.6522917Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\http\client.py", line 1277, in request
2022-06-25T08:31:15.6523467Z     self._send_request(method, url, body, headers, encode_chunked)
2022-06-25T08:31:15.6524016Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\http\client.py", line 1323, in _send_request
2022-06-25T08:31:15.6524561Z     self.endheaders(body, encode_chunked=encode_chunked)
2022-06-25T08:31:15.6525087Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\http\client.py", line 1272, in endheaders
2022-06-25T08:31:15.6525645Z     self._send_output(message_body, encode_chunked=encode_chunked)
2022-06-25T08:31:15.6526207Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\http\client.py", line 1032, in _send_output
2022-06-25T08:31:15.6526660Z     self.send(msg)
2022-06-25T08:31:15.6527201Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\http\client.py", line 972, in send
2022-06-25T08:31:15.6527658Z     self.connect()
2022-06-25T08:31:15.6528128Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\http\client.py", line 1439, in connect
2022-06-25T08:31:15.6528596Z     super().connect()
2022-06-25T08:31:15.6529055Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\http\client.py", line 944, in connect
2022-06-25T08:31:15.6529592Z     (self.host,self.port), self.timeout, self.source_address)
2022-06-25T08:31:15.6530134Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\socket.py", line 707, in create_connection
2022-06-25T08:31:15.6530671Z     for res in getaddrinfo(host, port, 0, SOCK_STREAM):
2022-06-25T08:31:15.6531198Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\socket.py", line 752, in getaddrinfo
2022-06-25T08:31:15.6532057Z     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
2022-06-25T08:31:15.6532696Z socket.gaierror: [Errno 11001] getaddrinfo failed
2022-06-25T08:31:15.6533043Z 
2022-06-25T08:31:15.6534152Z During handling of the above exception, another exception occurred:
2022-06-25T08:31:15.6534532Z 
2022-06-25T08:31:15.6534989Z Traceback (most recent call last):
2022-06-25T08:31:15.6535589Z   File "D:\a\_temp\4f3a0980-d178-4d00-93eb-2671f8d307aa.py", line 4, in <module>
2022-06-25T08:31:15.6536281Z     urllib.request.urlretrieve(url, path)
2022-06-25T08:31:15.6536921Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\urllib\request.py", line 247, in urlretrieve
2022-06-25T08:31:15.6537661Z     with contextlib.closing(urlopen(url, data)) as fp:
2022-06-25T08:31:15.6538265Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\urllib\request.py", line 222, in urlopen
2022-06-25T08:31:15.6538867Z     return opener.open(url, data, timeout)
2022-06-25T08:31:15.6539489Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\urllib\request.py", line 525, in open
2022-06-25T08:31:15.6540056Z     response = self._open(req, data)
2022-06-25T08:31:15.6540632Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\urllib\request.py", line 543, in _open
2022-06-25T08:31:15.6542341Z     '_open', req)
2022-06-25T08:31:15.6542921Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\urllib\request.py", line 503, in _call_chain
2022-06-25T08:31:15.6543488Z     result = func(*args)
2022-06-25T08:31:15.6544069Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\urllib\request.py", line 1393, in https_open
2022-06-25T08:31:15.6544714Z     context=self._context, check_hostname=self._check_hostname)
2022-06-25T08:31:15.6545339Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\urllib\request.py", line 1352, in do_open
2022-06-25T08:31:15.6545905Z     raise URLError(err)
2022-06-25T08:31:15.6546414Z urllib.error.URLError: <urlopen error [Errno 11001] getaddrinfo failed>
2022-06-25T08:31:15.6577359Z ##[error]The process 'C:\hostedtoolcache\windows\Python\3.7.9\x64\python.exe' failed with exit code 1
2022-06-25T08:31:15.6605697Z ##[section]Finishing: Download Miniforge

And the logs from conda-forge/scipy-feedstock#204:

2022-07-24T09:19:35.5266063Z ##[section]Starting: Download Miniforge
2022-07-24T09:19:35.5407288Z ==============================================================================
2022-07-24T09:19:35.5408980Z Task         : Python script
2022-07-24T09:19:35.5409427Z Description  : Run a Python file or inline script
2022-07-24T09:19:35.5409763Z Version      : 0.200.0
2022-07-24T09:19:35.5410090Z Author       : Microsoft Corporation
2022-07-24T09:19:35.5410480Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/python-script
2022-07-24T09:19:35.5411600Z ==============================================================================
2022-07-24T09:19:36.2032419Z [command]C:\hostedtoolcache\windows\Python\3.7.9\x64\python.exe D:\a\_temp\fe1e86ef-bf45-4fbb-8ddc-ceec86506bbe.py
2022-07-24T09:19:39.8710504Z Traceback (most recent call last):
2022-07-24T09:19:39.8713026Z   File "D:\a\_temp\fe1e86ef-bf45-4fbb-8ddc-ceec86506bbe.py", line 4, in <module>
2022-07-24T09:19:39.8714929Z     urllib.request.urlretrieve(url, path)
2022-07-24T09:19:39.8716712Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\urllib\request.py", line 276, in urlretrieve
2022-07-24T09:19:39.8718722Z     block = fp.read(bs)
2022-07-24T09:19:39.8719972Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\http\client.py", line 461, in read
2022-07-24T09:19:39.8721287Z     n = self.readinto(b)
2022-07-24T09:19:39.8722531Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\http\client.py", line 505, in readinto
2022-07-24T09:19:39.8723781Z     n = self.fp.readinto(b)
2022-07-24T09:19:39.8725056Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\socket.py", line 589, in readinto
2022-07-24T09:19:39.8726423Z     return self._sock.recv_into(b)
2022-07-24T09:19:39.8727873Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\ssl.py", line 1071, in recv_into
2022-07-24T09:19:39.8729080Z     return self.read(nbytes, buffer)
2022-07-24T09:19:39.8730347Z   File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\ssl.py", line 929, in read
2022-07-24T09:19:39.8732182Z     return self._sslobj.read(len, buffer)
2022-07-24T09:19:39.8733465Z ssl.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2570)
2022-07-24T09:19:39.8990604Z ##[error]The process 'C:\hostedtoolcache\windows\Python\3.7.9\x64\python.exe' failed with exit code 1
2022-07-24T09:19:39.9128099Z ##[section]Finishing: Download Miniforge

[maresb]

@h-vetinari I just added ssl.SSLError as a retryable exception, so it should now cover this latest case.

[jakirkham]

Wonder if we should just use a download tool instead of scripting this ourselves. For example Azure Pipelines has PowerShell, which could be used to download a file.

[maresb]

It's always nice to use something preexisting instead of scripting ourselves. However, I don't know any PowerShell. I'm happy to close this in favor of a better implementation, but I won't be able to write or test it. (I'm Linux-only.)