getting error "read: connection reset by peer" while downloading bpack deps

[gvolpe1968]

we are having recurring issues while downloading buildpacks dependencies from 2 specific regions : us-south, us-east
our logs indicates a connection reset issue as following examples :

2023-08-22 06:02:04 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-0-1/STG] Download [https://buildpacks.cloudfoundry.org/dependencies/dotnet-sdk/dotnet-sdk_6.0.408_linux_x64_any-stack_fad207e0.tar.xz] 2023-08-22 06:02:06 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-0-1/STG] **ERROR** Unable to install Dotnet SDK: read tcp 10.244.66.125:41976->104.118.246.84:443: read: connection reset by peer

2023-08-22 06:02:04 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-0-1/STG] Download [https://buildpacks.cloudfoundry.org/dependencies/dotnet-sdk/dotnet-sdk_6.0.408_linux_x64_any-stack_fad207e0.tar.xz] 
2023-08-22 06:02:06 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-0-1/STG] **ERROR** Unable to install Dotnet SDK: read tcp 10.244.66.125:41976->104.118.246.84:443: read: connection reset by peer

2023-08-22 05:37:52 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-3-2/STG] Download [https://buildpacks.cloudfoundry.org/dependencies/dotnet-sdk/dotnet-sdk_6.0.407_linux_x64_any-stack_f1497b98.tar.xz]
2023-08-22 05:37:55 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-3-2/STG] **ERROR** Unable to install Dotnet SDK: read tcp 10.245.203.234:36272->104.118.246.84:443: read: connection reset by peer

2023-08-22 06:22:02 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-1-1-0/STG] Download [https://buildpacks.cloudfoundry.org/dependencies/python/python_3.10.10_linux_x64_cflinuxfs3_73c718f0.tgz]
2023-08-22 06:22:03 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-1-1-0/STG] **ERROR** Unable to bootstrap python: read tcp 10.244.253.84:46256->104.118.246.84:443: read: connection reset by peer

after some more in deep investigations we noticed that your akamai service is limiting the download rate to 10MB returning error in case the download speed is higher
but this is specific for some ip addresses because when we connect from another availability zone the limit is to 100MB

in our case connecting from those US-EAST regional ip , AZ1 and AZ2 have the lower limit while AZ3 has the higher

AZ1 52.116.125.68 10MB/s
AZ2 169.59.191.31 10MB/s
AZ3 150.239.226.24 100MB/s

also connecting from US-SOUTH regional ip , AZ1 and AZ2 have the lower limit while AZ3 has the higher

AZ1 150.240.65.245 10MB/s
AZ2 150.239.168.150 10MB/s
AZ3 169.48.95.93 100MB/s

fyi this is the command we use for test from US-SOUTH

export cdn_ip=104.118.246.84; curl --limit-rate 50M -o x.xz https://buildpacks.cloudfoundry.org/dependencies/dotnet-sdk/dotnet-sdk_6.0.407_linux_x64_any-stack_f1497b98.tar.xz --resolve buildpacks.cloudfoundry.org:443:$cdn_ip

replace cdn_ip=104.96.236.92 for US-EAST

summarising we noticed a rate_limit to 10MB/s when connecting

to 
104.96.236.92
from 
52.116.125.68
169.59.191.31

and

to
104.118.246.84
from
150.240.65.245
150.239.168.150

[digarci13]

I perform the trace route command in 3 different cells in different our us-south env in AZ1,AZ2, and AZ3 and all of them I got the similar output

traceroute 104.118.246.84
traceroute to 104.118.246.84 (104.118.246.84), 30 hops max, 60 byte packets
 1  x.x.x.x (x.x.x.x)  0.118 ms  0.087 ms  0.065 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

ping 104.118.246.84
PING 104.118.246.84 (104.118.246.84) 56(84) bytes of data.
64 bytes from 104.118.246.84: icmp_seq=1 ttl=52 time=6.96 ms
64 bytes from 104.118.246.84: icmp_seq=2 ttl=52 time=6.82 ms
64 bytes from 104.118.246.84: icmp_seq=3 ttl=52 time=6.88 ms
64 bytes from 104.118.246.84: icmp_seq=4 ttl=52 time=6.80 ms
^C
--- 104.118.246.84 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 6.796/6.862/6.956/0.061 ms

I also installed mtr in one cell and this is the output:

diego-cell-us-south-1-0-0 (x.x.x.x) -> 104.118.246.84 (104.118.246.84)                                                                                                                2023-08-22T19:23:40+0000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                                                                                                             Packets               Pings
 Host                                                                                                                                                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. x.x.x.x                                                                                                                                                            0.0%    24    0.1   0.1   0.1   0.1   0.0
 2. (waiting for reply)
 3. (waiting for reply)
 4. (waiting for reply)
 5. (waiting for reply)
 6. (waiting for reply)
 7. (waiting for reply)
 8. (waiting for reply)
 9. (waiting for reply)
10. (waiting for reply)
11. (waiting for reply)
12. (waiting for reply)
13. (waiting for reply)
14. (waiting for reply)
15. 104.118.246.84                                                                                                                                                          0.0%    23    6.9   6.9   6.8   7.0   0.0

[digarci13]

Same thing in our us-east env:

# traceroute 104.96.236.92
traceroute to 104.96.236.92 (104.96.236.92), 30 hops max, 60 byte packets
 1  x.x.x.x (x.x.x.x)  0.192 ms  0.041 ms  0.019 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

ping 104.96.236.92
PING 104.96.236.92 (104.96.236.92) 56(84) bytes of data.
64 bytes from 104.96.236.92: icmp_seq=1 ttl=57 time=1.50 ms
64 bytes from 104.96.236.92: icmp_seq=2 ttl=57 time=1.62 ms
64 bytes from 104.96.236.92: icmp_seq=3 ttl=57 time=1.55 ms
64 bytes from 104.96.236.92: icmp_seq=4 ttl=57 time=1.59 ms
^C
--- 104.96.236.92 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 1.498/1.562/1.617/0.045 ms

[gvolpe1968]

We worked to create a new machine on a Cloud IaaS out of any VPC and tested with same results.
We also used some other VMs in other Infrastructure and still same issue.
so it is not limited to those source IPs i pasted before.
Can you pls ask CDN team if there is somehow a rate limit and to test the curl command being sure the download speed goes over 50MB/s (as an example) ?

[gvolpe1968]

another wave is currently happening on us-south region.

2023-08-23 12:49:27 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-4-1/STG] Download [https://buildpacks.cloudfoundry.org/dependencies/dotnet-sdk/dotnet-sdk_6.0.408_linux_x64_any-stack_fad207e0.tar.xz]
2023-08-23 12:49:27 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-4-1/STG] -----> Installing dotnet-sdk 6.0.408
2023-08-23 12:49:29 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-4-1/STG] **ERROR** Unable to install Dotnet SDK: read tcp 10.247.83.10:35410->104.118.246.84:443: read: connection reset by peer
2023-08-23 12:49:29 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-4-1/STG] Failed to compile droplet: Failed to run all supply scripts: exit status 15
2023-08-23 12:49:29 UTC [diego-cell-loggr-forwarder-agent/diego-cell-us-south-2-4-1/STG] Exit status 223

[dsboulder]

Ticket filed and going back and forth with akamai support