Skip to content

Commit

Permalink
Don't double-count internal telemetry (#409)
Browse files Browse the repository at this point in the history
We goofed when fiddling with the way we represent internal telemetry.
In the change we did three things:

  * change the aggregation of telemetry
  * remove zero-ing of the telemetry
  * issue zero points

The real problem comes when we issue the new aggregation, summing,
down to a sink that also performs aggregation. We're duplicating
values, potentially many times. Also, by issuing zero points we
increase aggregation load on the sink service for no real reason.

This commit takes us back to zero-ing and relies on the sink
aggregation to do the right thing. We also only emit a point if
it is non-zero.

Signed-off-by: Brian L. Troutwine <blt@postmates.com>
  • Loading branch information
blt authored Jan 31, 2018
1 parent f514a88 commit dac169c
Showing 1 changed file with 12 additions and 10 deletions.
22 changes: 12 additions & 10 deletions src/source/internal.rs
Original file line number Diff line number Diff line change
Expand Up @@ -70,16 +70,18 @@ pub fn report_full_telemetry(
macro_rules! atom_telem {
($name:expr, $atom:expr, $tags:expr, $chans:expr) => {
let now = time::now();
let value = $atom.load(Ordering::Relaxed);
let telem = Telemetry::new()
.name($name)
.value(value as f64)
.timestamp(now)
.kind(AggregationMethod::Sum)
.harden()
.unwrap()
.overlay_tags_from_map(&$tags);
util::send(&mut $chans, metric::Event::new_telemetry(telem));
let value = $atom.swap(0, Ordering::Relaxed);
if value != 0 {
let telem = Telemetry::new()
.name($name)
.value(value as f64)
.timestamp(now)
.kind(AggregationMethod::Sum)
.harden()
.unwrap()
.overlay_tags_from_map(&$tags);
util::send(&mut $chans, metric::Event::new_telemetry(telem));
}
}
}

Expand Down

0 comments on commit dac169c

Please sign in to comment.