How to use Microsoft.Testing.Extensions.Retry?

[McMlok]

Hello,

is there a documentation how to add retry logic to a test run? I added EnableMSTestRunner and OutputType=exe to my test project and also reference to Microsoft.Testing.Extensions.Retry. But one of my flaky test is still failing and I don't see any retry attempt. What I'm missing? Is there a way how to see that test retries?

My csproj is simple

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>net8.0;net9.0</TargetFrameworks>
    <WarningsAsErrors>Nullable</WarningsAsErrors>
     <EnableMSTestRunner>true</EnableMSTestRunner>
     <OutputType>Exe</OutputType>
  </PropertyGroup>
<ItemGroup>
  <PackageReference Include="MSTest" />
  <PackageReference Include="xunit" />
  <PackageReference Include="Microsoft.Testing.Extensions.Retry" />
</ItemGroup>
</Project>

And I run tests with dotnet test.

Thank you

[Evangelink]

You can have a look here https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-extensions-policy#retry. Please do let us know if there is something missing or unclear from the docs.

I see that you are referencing both xunit and MSTest in the same project. This is not a pattern we recommend (or enable) with the new platform.

[McMlok]

I have to upgrade xUnit to xUnit.V3 where is support to the new Test Platform. Instead I will have to rewrite 1k+ tests from xUnit to MSTest. And with it my test runs in Test Platform instead of VSTest, but when I enable retry I often got error with moving file.

=====================
Moving last attempt asset files to the default result directory
=====================

Moving file '......Tests_net8.0_x64.log' to '.....Tests_net8.0_x64.log'
Unhandled exception. System.IO.IOException: The process cannot access the file because it is being used by another process.
   at System.IO.FileSystem.MoveFile(String sourceFullPath, String destFullPath, Boolean overwrite)
   at Microsoft.Testing.Extensions.Policy.RetryOrchestrator.OrchestrateTestHostExecutionAsync() in /_/src/Microsoft.Testing.Extensions.Retry/RetryOrchestrator.cs:line 290
   at Microsoft.Testing.Platform.Hosts.TestHostOrchestratorHost.RunAsync() in /_/src/Platform/Microsoft.Testing.Platform/Hosts/TestHostOchestratorHost.cs:line 30
   at Microsoft.Testing.Platform.Builder.TestApplication.RunAsync() in /_/src/Platform/Microsoft.Testing.Platform/Builder/TestApplication.cs:line 240
   at Xunit.Runner.InProc.SystemConsole.TestingPlatform.TestPlatformTestFramework.RunAsync(String[] args, Action`2 extensionRegistration) in D:\a\xunit\xunit\src\xunit.v3.runner.inproc.console\TestingPlatform\TestPlatformTestFramework.cs:line 270
   at XunitAutoGeneratedEntryPoint.Main(String[] args) in D:\Packages\.nuget\xunit.v3.core\0.6.0-pre.7\_content\EntryPoint-mtp.cs:line 11

=== COMMAND LINE ===
C:\Program Files\dotnet\dotnet.exe exec ....Tests\bin\Debug\net8.0\....Tests.dll --internal-msbuild-node testingplatform.pipe.25150cb9d18d48c49b2451672f275548 --retry-failed-tests 1 --retry-failed-tests-max-tests 3  

But it could be a problem with xUnit.V3.

[Evangelink]

We don't yet have that many users of the retry functionality so it's also possible we have some latent bug there we didn't notice. Would you have some repro you could share with us?

[McMlok]

Sorry for late reply. I will try to prepare minimal repro because the project where I hit is on on-prem repository and is big. I added it to my TODO and I hope I will have time for it soon. For the project was better to rework flaky tests to not to be flaky. But still the new test platform will be better to use in the project it half test time from 5mins to 2.