Cannot send outbound requests using node fetch, failing with IPv6 DNS error UND_ERR_CONNECT_TIMEOUT

[undergroundwires]

Description

Sending more than one (sometimes two or three) requests using fetch results in DNS errors.

I get UND_ERR_CONNECT_TIMEOUT errors which is related to IPv6 issues in node.
Related issue for node: nodejs/node#41625,
Related issue for node fetch: nodejs/undic#1531
Related issue for GitHub runners not being able to do IPv6: #668, actions/runner#3138

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

Failed build: https://github.com/undergroundwires/node-fetch-ipv6/actions/runs/8350276298

Is it regression?

Yes, successful build runs are old

Expected behavior

The requests are being sent fine, this was the case before.

Actual behavior

I created a repository to reproduce this in as minimal way as possible:
https://github.com/undergroundwires/node-fetch-ipv6

I run fetch on using vite test runner on node and without any test runner. I get same results.
I realize that the first two requests are successful, I start getting the error after the second or third one.

See the test: test file

Second test (multiple fetch in order) fails, and all run fine locally. I run tests on default GitHub runners using both latest node 20 and 18. I get same errors:

Repro steps

1. Run the job from undergroundwires/node-fetch-ipv6
2. Check output

[Alexey-Ayupov]

Hello @undergroundwires, do I understand correctly that you are trying to fetch using ipv6?

[undergroundwires]

I @Alexey-Ayupov. According to my research, the error UND_ERR_CONNECT_TIMEOUT related to DNS, and community reports mostly about IPv6, that's why I brought up IPv6. I cannot relate this to any outbound request limitation, even second/third requests fails after 10 seconds interval.

The test failing is simple:

const urls = [
    'https://web.archive.org/web/20221029145712/https://kb.mozillazine.org/Downloads.rdf',
    'https://web.archive.org/web/20240120213614/https://techcommunity.microsoft.com/t5/windows-it-pro-blog/group-configuration-search-highlights-in-windows/ba-p/3263989',
    'https://web.archive.org/web/20230929132845/https://support.microsoft.com/en-us/topic/compatibility-update-for-keeping-windows-up-to-date-in-windows-server-2012-r2-and-windows-server-2008-r2-sp1-c62197fb-d711-f7d3-f135-172844b9f322',
    'https://web.archive.org/web/20231004100105/https://nvd.nist.gov/vuln/search/results?form_type=Basic&results_type=overview&query=microsoft+store&queryType=phrase&search_type=all&isCpeNameSearch=false',
    'https://web.archive.org/web/20240219205516/https://wiki.archlinux.org/title/spotify',
    'https://web.archive.org/web/20240119160347/https://github.com/undergroundwires/privacy.sexy/issues/247',
    'https://web.archive.org/web/20231206171559/https://bestgamingtips.com/fix-xbox-identity-provider-not-working/',
    'https://web.archive.org/web/20230806192800/https://www.hexacorn.com/blog/2018/09/02/beyond-good-ol-run-key-part-86/',
    'https://web.archive.org/web/20231020012236/https://answers.microsoft.com/es-es/windows/forum/all/windows-10-carpeta-y-archivos-sih/4d318121-fed6-4202-8b92-d4dc236b468e',
    'https://web.archive.org/web/20221029141626/https://kb.mozillazine.org/Places.sqlite'
  ];
  for (const url of urls) {
    console.log('Fetching: ', url);
    const response = await fetch(url, { method: 'HEAD'});
    await sleep(10000);
    expect(response.status).to.equal(200);
  }

The error I'm getting after first, sometimes second request:

Caused by: ConnectTimeoutError: Connect Timeout Error

[mikhailkoliada]

hosted runners do not support ipv6 yet, that is why every "real" ipv6 inquiry will fail, and there is no ETA on ipv6 fully available unfortunately.

[undergroundwires]

Hi @mikhailkoliada, I tested more and verify that this is not about IPv6 requests being made.

This is how I prove it's not about IPv6 resolution directly:

• It happens also when I set dns.setDefaultResultOrder('ipv4first'); which forces IPv4 resolution. I added that test to tests in the repository.
  Configuration: vite-test.spec.ts
• I tell ubuntu to prefer IPv4 through: sudo sh -c "echo 'precedence ::ffff:0:0/96 100' >> /etc/gai.conf"
  Configuration: .github/workflows/test.yaml
• I disable IPv6 on host container completely:

    sysctl -w net.ipv6.conf.all.disable_ipv6=1
    sysctl -w net.ipv6.conf.default.disable_ipv6=1
    sysctl -w net.ipv6.conf.lo.disable_ipv6=1
  
  

  Configuration: .github/workflows/test.yaml

I get UND_ERR_CONNECT_TIMEOUT regardless.

Can it be possible that this is because of some ongoing connection (network/DNS requests) limitation in the image or cloud service you're running?

[undergroundwires]

Here's my workaround (open-source and documented) that I hope that can help you too:

• privacy.sexy | force-ipv4: GitHub action to use for GitHub workflows.
• privacy.sexy | force-ipv4.sh.: Script for general use.

After days of research and trial/error, this is how I got this working:

1. Create a script called force-ipv4.sh, that configures system to prefer IPv4 over IPv6, call it to configure the machine. It was not easy to find a reliable cross-platform solution and I went Cloudflare WARP for DNS resolution along with some system configurations.
2. To easily use the script in GitHub workflows, create GitHub action called force-ipv4 and call it in GitHub runners.
3. Fixes the IPv6 request issues, and you can happily run e.g. fetch API from Node.

Related commit introducing this fix: undergroundwires/privacy.sexy@52fadcd