Skip to content

Commit

Permalink
Print the number of QuickCheck shrinks in the progress bar
Browse files Browse the repository at this point in the history
Currently, `tasty` and `tasty-quickcheck` will print a progress percentage, so
one can see test progression. However, once a property finds a failure and
starts shrinking, then test progression stops. Importantly, it is not clear how
the shrinker is progressing, though it would be useful information to show,
e.g., to judge whether a test is shrinking too slow, whether the shrinker loops,
or whether a shrunk test case hangs. This commit changes the progress bar to
print the number of shrinks in case of test failures. Succesful tests will still
print the progress percentage.
  • Loading branch information
jorisdral authored and Bodigrim committed Nov 5, 2024
1 parent d9d8cdf commit 7efa34a
Show file tree
Hide file tree
Showing 2 changed files with 20 additions and 4 deletions.
17 changes: 15 additions & 2 deletions quickcheck/Test/Tasty/QuickCheck.hs
Original file line number Diff line number Diff line change
Expand Up @@ -283,8 +283,21 @@ quickCheck yieldProgress args
= (.) (QC.quickCheckWithResult args)
$ QCP.callback
$ QCP.PostTest QCP.NotCounterexample
$ \QC.MkState {QC.maxSuccessTests, QC.numSuccessTests} _ ->
yieldProgress $ emptyProgress {progressPercent = fromIntegral numSuccessTests / fromIntegral maxSuccessTests}
$ \st@QC.MkState {QC.maxSuccessTests, QC.numSuccessTests} _ ->
yieldProgress $
if QC.numTotTryShrinks st > 0 then
emptyProgress {
progressText = showShrinkCount st
}
else
emptyProgress {
progressPercent = fromIntegral numSuccessTests / fromIntegral maxSuccessTests
}

-- Based on 'QuickCheck.Test.failureSummaryAndReason'.
showShrinkCount :: QC.State -> String
showShrinkCount st = show (QC.numSuccessShrinks st) ++ " shrink" ++ plural
where plural = if QC.numSuccessShrinks st == 1 then "" else "s"

successful :: QC.Result -> Bool
successful r =
Expand Down
7 changes: 5 additions & 2 deletions quickcheck/tests/test.hs
Original file line number Diff line number Diff line change
Expand Up @@ -110,8 +110,11 @@ main =
resultDescription =~ "Use .* to reproduce"

-- Run the test suite manually and check that progress does not go beyond 100%
, testProperty "Percent Complete" $ withMaxSuccess 1000 $ \(_ :: Int) -> ioProperty $ threadDelay 10000

, testProperty "Percent Complete" $
withMaxSuccess 1000 $ \(_ :: Int) -> ioProperty $ threadDelay 10000
, testProperty "Number of shrinks" $
expectFailure $ withMaxSize 1000 $ \(Large (x :: Int)) ->
ioProperty $ threadDelay 100000 >> pure (x <= 100)
]

run' :: Testable p => p -> IO Result
Expand Down

0 comments on commit 7efa34a

Please sign in to comment.