[8.x] PGO: update the existing benchmarks workflow to enable PGO builds (backport #13884) #14245
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![mergify[bot]](https://avatars.githubusercontent.com/in/10562?s=60&v=4){kind=small}
Add a benchmark workflow mode with automation to collect, preserve, and inject CPU profiles, enabling PGO builds. The new workflow will run on a schedule and raise a special pull request that includes the most recent representative CPU profile, which will be inserted as the `default.pgo` file into the main package and automatically used in the build pipeline. The actual schedule and the model for raising pull requests with updated profiles are subject to further revisions. This new workflow mode uses a lightweight output destination - a mock proxy (Moxy) from apm-perf to better isolate the performance component of the APM Server. (cherry picked from commit 5af8cf4)
|
I don't think that we need to backport this changes. |
2 tasks
1pkg
approved these changes
Oct 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation/summary
This PR implements changes outlined in #13859. It updates the existing benchmarks workflow to run standalone APM Server instance that produces a relevant CPU profile for PGO, then it copies, uploads and injects the obtained CPU profile into a PR, see example.
Benchmarks
The existing benchmarks results turned to be too unreliable to base PGO on. Because of the underlying dependency on ElasticSearch the difference in the throughput results could go above 10% from a workflow to workflow. The table below provides a view with the existing benchmarks results sample.
This all renders incremental PGO performance gains hard to observe and measure. Therefore, in this PR a new benchmark mode is introduced, which swaps ElasticSearch with a stubbed API http server (Moxy). Thus allowing us to better isolate and elevate APM Server performance component inside the benchmarks. The table below provides a view with the new isolated benchmarks results sample.
Using the benchmarks result sample data we can clearly observe that the results deviation for the new benchmark mode is in an order of magnitude lower in comparison to the existing ES based benchmarks. And now PGO performance improvements could be reliably observed.
The standalone APM Server benchmarks mode consists of running 3 separate EC2 instances in a VPC for apmbench, apm-server and moxy. Existing
benchmark_executorandstandalone_apm_serverterraform modules are reused and a similar new terraform modulemoxyis created.Results
PGO enabled builds show 5% performance gain on average across the standalone APM Server benchmarks workflow.
Checklist
For functional changes, consider:
How to test these changes
To observe and validate the changes please refer to the indexed PGO benchmarks results.
Related issues
#13859
This is an automatic backport of pull request #13884 done by [Mergify](https://mergify.com).