Provide historical summary of tests and PR comment on failed tests #567
Conversation
DiamondJoseph
force-pushed
the
test-tracking
branch
3 times, most recently
from
057127b to
529ea55
Compare
DiamondJoseph
requested review from
coretl,
callumforrester,
abbiemery and
jwlodek
|
Should the bot be commenting here? |
It can be configured to. Currently set to only comment if there are test failures, just to keep things slightly more foccussed: I wouldn't personally want a bot comment for every pushed commit. |
|
duration |
Github reckons 36s, so that part of the table definitely doesn't look right. And 440 tests, 1 skipped vs. 309/1 (do we have 131 paramterised tests options and they count them separate?). I'll look again at the ctrf export |
|
Pretty cool! Are there outlined criteria for describing a test as flaky? I.e. 2 failures out of 5 runs or something similar? Likewise, can it distinguish between temporarily broken tests on a dev PR and actually flaky tests? Also, not sure if possible, but would be cool if we have tests we know are flaky that fail to have the CI either ignore them or try running again so we don't need to merge PRs with failing CI because of a flaky test. |
|
I'd actively resist making the CI more resistant to flaky tests: if we're not being annoyed we won't prioritise fixing them. This is intended (to my mind) more as something for maintainers to monitor and if we see something become flaky making time to fix it. |
|
No idea how it decides what is what, hopefully we can find out :D |
|
@DiamondJoseph where did we get to on this? |
DiamondJoseph
force-pushed
the
test-tracking
branch
from
529ea55 to
e3d5d7e
Compare
|
I've just switched it to always report on test status, there may be insight in historical test data there that might be missed on just the fails(?) and rebased it. I think we have to decide whether to enable or not and then potentially revisit after it's been enabled for a while. |
Test Summary - Run #2273
🎉 All tests passed! |
2 similar comments
Test Summary - Run #2273
🎉 All tests passed! |
Test Summary - Run #2273
🎉 All tests passed! |
|
Hmm definitely not a fan of putting it in at the moment with 3 reports like that |
Makes tracking e.g. #23 easier, may enable catching other issues?
See the summary page of this PR's Checks section.
Provides table of test behaviour over the previous N (default 10) runs. Untested but should also comment on any PRs with failed tests: example here from the action itself.
I'd prefer if it was a configured github action rather than running
npx <some command>...I'd like to try this out here and on some other repos for a Diamond Copier issue