Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test benchmarks in CI #16987

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Test benchmarks in CI #16987

wants to merge 3 commits into from

Conversation

BD103
Copy link
Member

@BD103 BD103 commented Dec 26, 2024

Objective

  • This reverts Remove benchmarks from CI test #16833, and completely goes against Do not test benchmarks in CI #16803.
  • Turns out running cargo test --benches runs each benchmark once, without timing it, just to ensure nothing panics. This is actually desired because we can use it to verify benchmarks are working correctly without all the time constraints of actual benchmarks.

Solution

  • Add the --benches flag to the CI test command.

Testing

  • cargo run -p ci -- test

@BD103 BD103 added D-Trivial Nice and easy! A great choice to get started with Bevy A-Build-System Related to build systems or continuous integration C-Testing A change that impacts how we test Bevy or how users test their apps C-Benchmarks Stress tests and benchmarks used to measure how fast things are S-Needs-Review Needs reviewer attention (from anyone!) to move forward labels Dec 26, 2024
@BenjaminBrienen
Copy link
Contributor

Something happened with the bench

thread 'main' panicked at benches/benches/bevy_tasks/iter.rs:80:25:

@@ -17,7 +17,8 @@ impl Prepare for TestCommand {
vec![PreparedCommand::new::<Self>(
cmd!(
sh,
"cargo test --workspace --lib --bins --tests {no_fail_fast}"
// `--benches` runs each benchmark once, to verify that they behave correctly.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// `--benches` runs each benchmark once, to verify that they behave correctly.
// `--benches` runs each benchmark once in order to verify that they behave correctly. (Don't panic)

@BenjaminBrienen
Copy link
Contributor

Approving since the failure should be fixed in a separate PR

Integer overflow panics in debug mode! Let's make it explicitly wrapping, just like how it is in release mode.
@BD103
Copy link
Member Author

BD103 commented Dec 26, 2024

Approving since the failure should be fixed in a separate PR

Surprisingly not! The panic was an integer overflow that was never caught in our benchmarks because they are usually compiled in release mode. I fixed it in aa2e600, though, to explicitly wrap.

@BD103 BD103 added S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it and removed S-Needs-Review Needs reviewer attention (from anyone!) to move forward labels Dec 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Build-System Related to build systems or continuous integration C-Benchmarks Stress tests and benchmarks used to measure how fast things are C-Testing A change that impacts how we test Bevy or how users test their apps D-Trivial Nice and easy! A great choice to get started with Bevy S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants