Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability for skip to report a message saying why a test is skipped #153

Open
nickrobinson251 opened this issue Apr 11, 2024 · 1 comment
Open
Labels
speculative a feature idea that we are undecided about

Comments

@nickrobinson251
Copy link
Collaborator

we have code that uses tags on testitems with a filter function passed to runtests in order to skip tests e.g.

@testitem "foo" tags=[:recmod_incompat] begin
   # tests here
end
function check_recovery_mode(test_item)
    should_run = true
    if RECOVERY_MODE()
        should_run = !(:recmode_incompat in test_item.tags)
    end
    if !should_run
        @warn "Skipping test `$(test_item.name)` because it is \
        incompatible with recovery mode."
    end
    return should_run
end
runtests(check_recovery_mode, ...)

This is preferred because it allows printing a message about why a testitem is being skipped, which is important when there are multiple conditions e.g.

@testitem "foo" tags=[:recmod_incompat, gapmode_incompat] begin
   # tests here
end

here we'd log the testitem is skipped because of recmode_incompat, whereas if we'd used skip like

@testitem "foo" skip=:(RECOVERY_MODE() || GAP_MODE()) begin
   # tests here
end

we would not know which condition were true.

@nickrobinson251
Copy link
Collaborator Author

a suggestion from @charnik:

We could drop the extra function in runtests() if we can setup ReTestItems such that it knows which tags to skip and the reason why. Like passing a dictionary skip_tags = Dict(:recmode_incompat => "Skip reason") or setting an environment variable RETESTITEMS_SKIPTAGS="recmode_incompat:\"Skip reason\"; gapmode_incompat:\"Another skip reason\""

@nickrobinson251 nickrobinson251 added the new feature A feature we would like to add label May 3, 2024
@nickrobinson251 nickrobinson251 added speculative a feature idea that we are undecided about and removed new feature A feature we would like to add labels Sep 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
speculative a feature idea that we are undecided about
Projects
None yet
Development

No branches or pull requests

1 participant