Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests pass, but it takes several manual retries (intermittent failures) #1058

Open
moreati opened this issue Apr 9, 2024 · 3 comments
Open

Comments

@moreati
Copy link
Member

moreati commented Apr 9, 2024

Exact causes are unknown. Linux+Ansible tests are most susceptible.

To re-run tests for a PR add a comment containing

/AzurePipelines run

(from https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/github?view=azure-devops&tabs=yaml#comment-triggers)

Originally posted by @moreati in #1054 (comment)

Failures are a mixture of timeouts and failing to find expected text in logging output.

@moreati moreati changed the title Tests pass, but it takes serveral retries. The problem predates adding Ansible 9, but it's getting worse. Causes unknown. Linux+Ansible tests are most susceptible. Tests pass, but it takes serveral manual retries Apr 9, 2024
@moreati
Copy link
Member Author

moreati commented Apr 12, 2024

HTTP GET to httpbin.org sometimes times out. Probably best to eliminate the external dependency.

moreati added a commit to moreati/mitogen that referenced this issue Apr 12, 2024
Should have higher uptime, and make us less of a burden. Refs mitogen-hq#1058
@moreati moreati changed the title Tests pass, but it takes serveral manual retries Tests pass, but it takes sereral manual retries May 8, 2024
rjhesketh pushed a commit to bbc/mitogen that referenced this issue Jul 10, 2024
Should have higher uptime, and make us less of a burden. Refs mitogen-hq#1058
@moreati moreati changed the title Tests pass, but it takes sereral manual retries Tests pass, but it takes several manual retries (intermittent failures) Sep 12, 2024
@moreati moreati pinned this issue Sep 12, 2024
@moreati
Copy link
Member Author

moreati commented Sep 26, 2024

https://github.com/mitogen-hq/mitogen/actions/runs/11051662112/job/30702035104?pr=1080 confirms that the Github Actions workflow works with third-party PRs. The Action is able to pull the container images.

moreati added a commit to moreati/mitogen that referenced this issue Oct 6, 2024
These targets are not used by any active tests, and the large numbers of hosts
multiply the size of the taskvars disctionary in memory to many (10s) MiB.

refs mitogen-hq#1058
moreati added a commit to moreati/mitogen that referenced this issue Oct 6, 2024
These targets are not used by any active tests, and the large numbers of hosts
multiply the size of the taskvars disctionary in memory to many (10s) MiB.

refs mitogen-hq#1058
moreati added a commit to moreati/mitogen that referenced this issue Oct 10, 2024
This reduces the number of jobs from 48 to 24. The Mitogen part of the test
suite has been parameterized on the Linux container targets to be run against.
Both the Ansible tests & Mitogen tests now use the same source of truth to
control which targets to use: environment variable MITOGEN_TEST_DISTRO_SPECS.
This replaces the two mutually exclusive env vars DISTRO and DISTROS. I've
also removed vestgial traces of an unused env var MITOGEN_TEST_DISTRO.

Parameterization adapted from
https://eli.thegreenplace.net/2014/04/02/dynamically-generating-python-test-cases

refs mitogen-hq#1058, mitogen-hq#1059
moreati added a commit to moreati/mitogen that referenced this issue Oct 10, 2024
This reduces the number of jobs from 48 to 24. The Mitogen part of the test
suite has been parameterized on the Linux container targets to be run against.
Both the Ansible tests & Mitogen tests now use the same source of truth to
control which targets to use: environment variable MITOGEN_TEST_DISTRO_SPECS.
This replaces the two mutually exclusive env vars DISTRO and DISTROS. I've
also removed vestgial traces of an unused env var MITOGEN_TEST_DISTRO.

Parameterization adapted from
https://eli.thegreenplace.net/2014/04/02/dynamically-generating-python-test-cases

refs mitogen-hq#1058, mitogen-hq#1059
@moreati
Copy link
Member Author

moreati commented Nov 7, 2024

New contender #1185: BlockingIOError: [Errno 35] Resource temporarily unavailable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant