Releases: Badbond/jacoco
math-min
Description
This version of JaCoCo adds the functionality to track line execution count. Essentially, we have changed the underlying data structure from a boolean[] to a int[] and increment values on every execution. We then cap the values in this int[] by invoking Math.min along with value Integer.MAX_VALUE - 1. This implementation is derived from JaCoCo version v0.8.7.
Implementation details
We have changed the bytecode injected in instrumented classes to hold an int[] instead of a boolean[] and then for its probes inject the bytecode shown below instead of that described in JaCoCo documentation. Please note that we then invoke Math.min with the incremented value along with Integer.MAX_VALUE- 1 which then prevents the integer from overflowing on next increment.
ALOAD <array>
xPUSH <probeid>
DUP2
IALOAD
ICONST_1
IADD
LDC_2147483646
INVOKESTATIC java/lang/Math.min(II)I
IASTORE
The Java equivalent of this code is: array[probeid] = Math.min(array[probeid]++, Integer.MAX_VALUE - 1)`` instead of array[index] = true`. Next to the added bytecode instructions, the maximum stack size has increased from 3 to 4.
The execution counts are reported only on source code lines (see caveats below) in XML and HTML reporting. The XML reporting exposes a new attribute called ec (execution count) and the HTML reporting shows the execution count of a line once hovered.
Performance
For our performance benchmarking, we have performed 20 local Maven builds running tests of a different application with the Maven goals jacoco:prepare-agent and jacoco:report executed. We compared performance against version v0.8.7 (which this code is based upon). These benchmarks have shown, using a Tukey-Kramer test after validating its assumptions, that we are 95% confident performance has decreased between ~3.2% and ~4.6%. However, this performance decrease might be different on other projects or in other environments.
Caveats
We made some decisions in our implementation that came with some caveats. These caveats are described below.
- This implementation stops registered executions once it has reached
Integer.MAX_VALUE - 1and therefore can skew the results from the analysis. This can be mitigated by dumping data often and resetting the probe array. - As JaCoCo maps instruction execution to source code lines, we can not determine on a statement level how often a statement has been executed. The heuristic that we chose to implement it to take the maximum number of executions of all instruction on a single line and report that. We mitigate this threat to validity based on the fact that most code is formatted such that statements are spread out over multiple lines.
- The implementation of the injected bytecode that updates the execution data is not thread safe. Synthetic tests show that with such an implementation, 12 threads on a 12 virtual core system misses more than half of the executions when they all update a single value in a array. For our situation, we consider this threat mitigated as this will likely not occur as much in a real scenario (different start times, different execution paths) nor will it impact the resulting analysis too much if some executions were not tracked. However, for your situation this might differ.
- Testing has changed to a minimal level to make the build pass. Therefore, most of them are based on executions counts of 0 or 1 (executed or not). Some tests have been altered to handle larger execution counts (e.g. when merging instructions). However, this test coverage is not high enough to have the functionality tested with confidence.
- As this fork is to be used in the dynamic analysis of a project as part of a master thesis, we have applied some changes that are specific to this thesis. The biggest change is that we do not ignore generated classes by
org.immutables.value.Generatedandjavax/annotation/processing/Generated. - This has been implemented for a specific scenario. Please use at your own risk (as enforced by license).
long-overflow
Description
This version of JaCoCo adds the functionality to track line execution count. Essentially, we have changed the underlying data structure from a boolean[] to a long[] and increment values on every execution. This implementation is derived from JaCoCo version v0.8.7.
Implementation details
We have changed the bytecode injected in instrumented classes to hold an long[] instead of a boolean[] and then for its probes inject the following bytecode instead of that described in JaCoCo documentation:
ALOAD <array>
xPUSH <probeid>
DUP2
LALOAD
LCONST_1
LADD
LASTORE
The Java equivalent of this code is: array[index]++ instead of array[index] = true. Next to the added bytecode instructions, the maximum stack size has increased from 3 to 4.
The execution counts are reported only on source code lines (see caveats below) in XML and HTML reporting. The XML reporting exposes a new attribute called ec (execution count) and the HTML reporting shows the execution count of a line once hovered.
Performance
For our performance benchmarking, we have performed 20 local Maven builds running tests of a different application with the Maven goals jacoco:prepare-agent and jacoco:report executed. We compared performance against version v0.8.7 (which this code is based upon). These benchmarks have shown, using a Tukey-Kramer test after validating its assumptions, that we are 95% confident performance has decreased between ~1.1% and ~2.5%. However, this performance decrease might be different on other projects or in other environments.
Note that we can not state that this version significantly differs in performance from that of integer-overflow.
Caveats
We made some decisions in our implementation that came with some caveats. These caveats are described below.
- This implementation is suspectible to long overflowing. Therefore, when code is executed more than
Long.MAX_VALUEtimes, it will report negative execution counts (and therefore consider lines uncovered) or once it passes 0, it will document wrong execution counts. This can be mitigated by dumping data often and resetting the probe array. - As JaCoCo maps instruction execution to source code lines, we can not determine on a statement level how often a statement has been executed. The heuristic that we chose to implement it to take the maximum number of executions of all instruction on a single line and report that. We mitigate this threat to validity based on the fact that most code is formatted such that statements are spread out over multiple lines.
- The implementation of the injected bytecode that updates the execution data is not thread safe. Synthetic tests show that with such an implementation, 12 threads on a 12 virtual core system misses more than half of the executions when they all update a single value in a array. For our situation, we consider this threat mitigated as this will likely not occur as much in a real scenario (different start times, different execution paths) nor will it impact the resulting analysis too much if some executions were not tracked. However, for your situation this might differ.
- Testing has changed to a minimal level to make the build pass. Therefore, most of them are based on executions counts of 0 or 1 (executed or not). Some tests have been altered to handle larger execution counts (e.g. when merging instructions). However, this test coverage is not high enough to have the functionality tested with confidence.
- As this fork is to be used in the dynamic analysis of a project as part of a master thesis, we have applied some changes that are specific to this thesis. The biggest change is that we do not ignore generated classes by
org.immutables.value.Generatedandjavax/annotation/processing/Generated. - This has been implemented for a specific scenario. Please use at your own risk (as enforced by license).
integer-overflow
Description
This version of JaCoCo adds the functionality to track line execution count. Essentially, we have changed the underlying data structure from a boolean[] to an int[] and increment values on every execution. This implementation is derived from JaCoCo version v0.8.7.
Implementation details
We have changed the bytecode injected in instrumented classes to hold an int[] instead of a boolean[] and then for its probes inject the following bytecode instead of that described in JaCoCo documentation:
ALOAD <array>
xPUSH <probeid>
DUP2
IALOAD
ICONST_1
IADD
IASTORE
The Java equivalent of this code is: array[index]++ instead of array[index] = true. Next to the added bytecode instructions, the maximum stack size has increased from 3 to 4.
The execution counts are reported only on source code lines (see caveats below) in XML and HTML reporting. The XML reporting exposes a new attribute called ec (execution count) and the HTML reporting shows the execution count of a line once hovered.
Performance
For our performance benchmarking, we have performed 20 local Maven builds running tests of a different application with the Maven goals jacoco:prepare-agent and jacoco:report executed. We compared performance against version v0.8.7 (which this code is based upon). These benchmarks have shown, using a Tukey-Kramer test after validating its assumptions, that we are 95% confident performance has decreased between ~0.9% and ~2.3%. However, this performance decrease might be different on other projects or in other environments.
Note that we can not state that this version significantly differs in performance from that of long-overflow.
Caveats
We made some decisions in our implementation that came with some caveats. These caveats are described below.
- This implementation is suspectible to integer overflowing. Therefore, when code is executed more than
Integer.MAX_VALUEtimes, it will report negative execution counts (and therefore consider lines uncovered) or once it passes 0, it will document wrong execution counts. This can be mitigated by dumping data often and resetting the probe array. - As JaCoCo maps instruction execution to source code lines, we can not determine on a statement level how often a statement has been executed. The heuristic that we chose to implement it to take the maximum number of executions of all instruction on a single line and report that. We mitigate this threat to validity based on the fact that most code is formatted such that statements are spread out over multiple lines.
- The implementation of the injected bytecode that updates the execution data is not thread safe. Synthetic tests show that with such an implementation, 12 threads on a 12 virtual core system misses more than half of the executions when they all update a single value in a array. For our situation, we consider this threat mitigated as this will likely not occur as much in a real scenario (different start times, different execution paths) nor will it impact the resulting analysis too much if some executions were not tracked. However, for your situation this might differ.
- Testing has changed to a minimal level to make the build pass. Therefore, most of them are based on executions counts of 0 or 1 (executed or not). Some tests have been altered to handle larger execution counts (e.g. when merging instructions). However, this test coverage is not high enough to have the functionality tested with confidence.
- As this fork is to be used in the dynamic analysis of a project as part of a master thesis, we have applied some changes that are specific to this thesis. The biggest change is that we do not ignore generated classes by
org.immutables.value.Generatedandjavax/annotation/processing/Generated. - This has been implemented for a specific scenario. Please use at your own risk (as enforced by license).
double-cast
Description
This version of JaCoCo adds the functionality to track line execution count. Essentially, we have changed the underlying data structure from a boolean[] to a int[] and increment values on every execution. We then cap the values in this int[] by incrementing the value in a casted double and then casting back to an int. This implementation is derived from JaCoCo version v0.8.7.
Implementation details
We have changed the bytecode injected in instrumented classes to hold an int[] instead of a boolean[] and then for its probes inject the bytecode shown below instead of that described in JaCoCo documentation. Please note that we cast the value in the array to a double, then increment this double, and then cast it back to an integer to prevent integer overflowing.
ALOAD <array>
xPUSH <probeid>
DUP2
IALOAD
I2D
DCONST_1
DADD
D2I
IASTORE
The Java equivalent of this code is: array[probeid] = (int) (array[probeid] + (double) 1) instead of array[index] = true. Next to the added bytecode instructions, the maximum stack size has increased from 3 to 6 (as double is a 64-bit structure taking up 2 stacks and we have two doubles at most on the stack).
The execution counts are reported only on source code lines (see caveats below) in XML and HTML reporting. The XML reporting exposes a new attribute called ec (execution count) and the HTML reporting shows the execution count of a line once hovered.
Performance
For our performance benchmarking, we have performed 20 local Maven builds running tests of a different application with the Maven goals jacoco:prepare-agent and jacoco:report executed. We compared performance against version v0.8.7 (which this code is based upon). These benchmarks have shown, using a Tukey-Kramer test after validating its assumptions, that we are 95% confident performance has decreased between ~2.0% and ~3.4%. However, this performance decrease might be different on other projects or in other environments.
Caveats
We made some decisions in our implementation that came with some caveats. These caveats are described below.
- This implementation stops registered executions once it has reached
Integer.MAX_VALUEand therefore can skew the results from analysis. This can be mitigated by dumping data often and resetting the probe array. - As JaCoCo maps instruction execution to source code lines, we can not determine on a statement level how often a statement has been executed. The heuristic that we chose to implement it to take the maximum number of executions of all instruction on a single line and report that. We mitigate this threat to validity based on the fact that most code is formatted such that statements are spread out over multiple lines.
- The implementation of the injected bytecode that updates the execution data is not thread safe. Synthetic tests show that with such an implementation, 12 threads on a 12 virtual core system misses more than half of the executions when they all update a single value in a array. For our situation, we consider this threat mitigated as this will likely not occur as much in a real scenario (different start times, different execution paths) nor will it impact the resulting analysis too much if some executions were not tracked. However, for your situation this might differ.
- Testing has changed to a minimal level to make the build pass. Therefore, most of them are based on executions counts of 0 or 1 (executed or not). Some tests have been altered to handle larger execution counts (e.g. when merging instructions). However, this test coverage is not high enough to have the functionality tested with confidence.
- As this fork is to be used in the dynamic analysis of a project as part of a master thesis, we have applied some changes that are specific to this thesis. The biggest change is that we do not ignore generated classes by
org.immutables.value.Generatedandjavax/annotation/processing/Generated. - This has been implemented for a specific scenario. Please use at your own risk (as enforced by license).