[DRAFT] Changelog types #4427
Closed
[DRAFT] Changelog types #4427
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^Benchmark_Set$ github.com/aquasecurity/tracee/pkg/changelog -benchtime=10000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/changelog cpu: AMD Ryzen 9 7950X 16-Core Processor Benchmark_Set/All_Scenarios-32 10000000 1337 ns/op 992 B/op 42 allocs/op Benchmark_Set/Within_Limit-32 10000000 2130 ns/op 3424 B/op 60 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/changelog 350.043s
The current Changelog structure consumes a significant amount of memory due to the allocation of metadata for each field/instance. As the number of fields increases, the memory usage grows linearly. Approximately 240 bytes per field were observed just for metadata, excluding the actual data and pointers for each field. To reduce memory consumption, the new changelog.Entries[T] implementation uses a slice of slices approach to store instances instead of storing metadata for each field separately. --- | Caches | GOGC | Branch | *Heap Use | *Heap | Diff of | Proctree | | | | | (Avg) | Growth | main | | |--------|------|--------|--------- -|------- -|---------|----------| | - | 5 | main | 18 | - | - | off | | 16384 | 5 | main | 125 | 107 | - | on | | 32768 | 5 | main | 209 | 191 | - | on | |---------------------------------- --------- ----------------------| | - | 5 | new | 18 | - | - | off | | 16384 | 5 | new | 81 | 63 | -41.12% | on | | 32768 | 5 | new | 118 | 100 | -47.64% | on | * in MB With GOGC set to 5, the new implementation reduces average heap usage by approximately 41% when using cache sizes of 16,384. For cache sizes of 32,768, the reduction is around 47%. The "Heap Use" and "Heap Growth" columns serve as a good indicator of memory consumption and can assist in determining optimal cache sizes. --- The Set method was hugely improved, reducing the number of allocations, memory usage, and execution time. The benchmark results are as follows: Running tool: /home/gg/.goenv/versions/1.22.4/bin/go test -benchmem -run=^$ -tags ebpf -bench ^Benchmark_Set$ github.com/aquasecurity/tracee/pkg/changelog -benchtime=10000000x goos: linux goarch: amd64 pkg: github.com/aquasecurity/tracee/pkg/changelog cpu: AMD Ryzen 9 7950X 16-Core Processor Benchmark_Set/All_Scenarios-32 10000000 444.4 ns/op 224 B/op 3 allocs/op Benchmark_Set/Within_Limit-32 10000000 650.2 ns/op 992 B/op 5 allocs/op PASS ok github.com/aquasecurity/tracee/pkg/changelog 367.236s | Scenario | ns/op. (%) | B/op Reduc. (%) | Alloc. Reduc. (%) | |-------------------|------------|-----------------|-------------------| | **All Scenarios** | -66.8% | -77.4% | -92.9% | | **Within Limit** | -69.5% | -71.0% | -91.7% |
|
To be continued after #4384 merged. |
|
Implemented back in #4384 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
1. Explain what the PR does
2. Explain how to test it
3. Other comments