TestReports.test() fails to act as Pkg.test() on julia 1.8.0

[BambOoxX]

I'm currently trying to generate unit test reports for CI.
This means replacing occurrences of Pkg.test() with TestReports.test() as advertised in https://juliatesting.github.io/TestReports.jl/stable/manual/#Testing-Packages.
However, when using a test/runtests.jl test script along with a dedicated Project.toml and Manifest.toml containing TestReports as a dependency, there are some issues I do not understand so far.

• While doing (<MyPackage>) pkg>test will lauch the test suite from my package's environment, I can"t figure how to do the same for TestReports.test()
• If I add TestReports to my package's dependencies (which does not seem a julia thing to do I guess) doing using TestReports; TestReports.test() succeeds
• If I add TestReports to my package's test environment, doing TestReports.test("MyPackage") fails due to the package not being installed (which seems totally normal).
• If I do using TestReports in my global environment then activate my package's environment and do TestReports.test() I get the following message

ERROR: MyPackage has version 1.1.0 of TestReports in its test manifest
which is not compatible with the version being used by TestReports (1.0.0)
Update the compat entry of MyPackage.
This message looks somewhat legit and seems to assume TestReports should be in my package's dependencies, although the version numbers reported seem to be nonsense since the latest version TestReports is v 0.7.1 so far.

To sum this up, is there a way to use TestReports without depending on it directly ?

[mmiller-max]

To sum this up, is there a way to use TestReports without depending on it directly ?

Because TestReports is not in Base it does have to be added to an environment to be used, the best way depends on whether you are doing it in CI or locally.

In CI I think the best way to do this is to not have TestReports as a test dependency and do

Julia --project=/dir/of/MyPackage -e 'using Pkg; Pkg.add("TestReports"); using TestReports; TestReports.test("MyPackage")'

Locally, I would have TestReports in my global environment (@1.8) and not in the dependencies or test dependencies of my package. You won't see the error above if TestReports is just in your global environment.

ERROR: MyPackage has version 1.1.0 of TestReports in its test manifest
which is not compatible with the version being used by TestReports (1.0.0)
Update the compat entry of MyPackage.
This message looks somewhat legit and seems to assume TestReports should be in my package's dependencies, although the version numbers reported seem to be nonsense since the latest version TestReports is v 0.7.1 so far.

I think there is a mistake here - It shouldn't say the the first TestReports, instead it should give you the dependency name. I think I've fixed this on branch mm/correct-manifest-dep-check. Could you try that branch and see what the error says please?

Also, I'm not sure these checks are working properly after looking at them just now!

[BambOoxX]

On branch mm/correct-manifest-dep-check now I get

ERROR: myPkg has version 1.1.0 of EzXML in its test manifest
which is not compatible with the version being used by TestReports (1.0.0)
Update the compat entry of myPkg.

But myPkg hat no mention of EzXML in its Manifest.toml

[mmiller-max]

On branch mm/correct-manifest-dep-check now I get

ERROR: myPkg has version 1.1.0 of EzXML in its test manifest
which is not compatible with the version being used by TestReports (1.0.0)
Update the compat entry of myPkg.

But myPkg hat no mention of EzXML in its Manifest.toml

Is it possible to share your package Project.toml/Manifest.toml to help debugging?

[BambOoxX]

Sorry for the long delay. I cannot share the full Project.toml/Manifest.toml but basically

• EzXML is only present in my main environment, not my package's dependencies, not it's test dependencies
• My package does not rely on any package relying on EzXML (no trace of it in the Manifest)
• Only the test environment contains EzXML (v1.1.0) through LocalCoverage, that my package test environment uses.

EDIT : Bluntly removing LocalCoverage to really have no trace of EzXML removes the error. So it seems the manifest that leads to the error is really the one of the test environment and not the one of the package itself

[mmiller-max]

Thanks for this, I'll add LocalCoverage to an environment and try to get this working

[mmiller-max]

Perhaps this is an issue with the global environment, is the version of EzXML installed in that 1.0.0? If so you should be able to update it to 1.1.0 and then it should work.

[BambOoxX]

Hi, I just checked the Manifest of my global environment but there were only indirect references to EzXML v1.1.0. I still got the same error at that point.
Still, for some reason I added then removed EzXML to my global environment, then retried. And now it seems to work (there was some Pkg.gc() that was triggered).

Note that the test run now fails with

Got exception outside of a @test
LoadError: MethodError: no method matching ReportingTestSet(::String; verbose::Bool)