Flaky smoke tests and performance issues #1442
Replies: 4 comments 6 replies
-
|
I would like feedback from @mtcherni95, @itaysk and @yanivagman in this topic, if possible. For the smoke tests case, @danielpacak and I can select a single signature and a single event (ptrace + anti_debugging_ptraceme.rego) and, by filtering amount of events, make the smoke test to be very trustful (even when github node is loaded). I'm more worried about the general usage tbh. |
Beta Was this translation helpful? Give feedback.
-
|
A quick example to illustrate the reduced throughput in between tests (1) and (2). By using the "pipe viewer" tool, I'm able to see the throughput in the pipe.
The speed tops at ~1.9MiB/s in this test box.
The speed in between tracee-ebpf and tracee-rules tops at ~7MiB/s. If we have a big enough circular buffer acting like FIFO in between the |
Beta Was this translation helpful? Give feedback.
-
|
Alright, I played with this idea a little bit (still using pipe and not changing tracee)... I have created a #include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>
#include <glib.h>
#include <glib/gprintf.h>
#define chunk_size 64 * 1024 * 1024
GAsyncQueue *queue;
volatile gint allocated;
void *readstdin(void *ptr) {
int multi = 1000;
gchar *entry;
for(;;) {
entry = g_malloc0(chunk_size);
read(fileno(stdin), entry, chunk_size);
g_async_queue_push(queue, entry);
if ((multi % 1000) == 0) {
fprintf(stderr, "buffer size: %d\n", g_async_queue_length(queue));
fflush(stderr);
}
multi++;
}
}
void *writestdout(void *ptr) {
gchar *entry;
for(;;) {
entry = g_async_queue_pop(queue);
g_printf("%s", (gchar *) entry);
fflush(stdout);
g_free(entry);
}
}
int main(int arc, char **argv)
{
pthread_t read_t, write_t;
int ret = 0;
allocated = 0;
queue = g_async_queue_new();
ret |= pthread_create(&read_t, NULL, readstdin, NULL);
ret |= pthread_create(&write_t, NULL, writestdout, NULL);
if (ret != 0) {
fprintf(stdout, "error creating threads\n");
exit(1);
}
pthread_join(read_t, NULL);
pthread_join(write_t, NULL);
return 0;
}I haven't lost eBPF events while caching (of course there is an OS limit for the cache size) and I haven't lost any TRC-2 signature detection during "bursts" of the test case ( Of course, because of caching, the detection sometimes happen much later than the time the |
Beta Was this translation helpful? Give feedback.
-
|
@mtcherni95 So, summarizing, a big cache in between Up to 8 MB/s of data is being produced with the selected events, and around 2MB/s being consumed for multiple signatures loaded. The delta is what defines the size of the buffer we need (for this particular case). Considering a sustained workload (of this test case), for X amount of seconds, will define the buffer size not to lose detections (and OS limits as well). Watermarks of this buffer usage might define event priority reductions (so the input throughput is reduced and output can catchup if buffer wasn't enough). We can have classes of events, and start reducing priorities per class when buffer watermark reaches 50%, 60, 70 etc. Not sure this is the direction you are taking, just thinking about it a bit more. Note: removing the OS file handling logic (and data copying from user land to kernel and back) will also help the throughput but the ratio 4:1 in production: consumption will, likely, remain. |
Beta Was this translation helpful? Give feedback.
-
@NDStrahilevitz please join us in the discussion here (your take is much appreciated). Yes, so... we would have to calculate the buffer size formula based on:
The heaviest workload possible has to be benchmarked depending on the host size. We have to pick a workload (traced by tracee) that maximizes My proposal is that, instead of measuring amount of lost events (when things went south already), we change a priority scale that reduces the amount of events being sent whenever we reach different buffer watermarks (50%, 60%, 70%). And, whenever buffer is brought to lower marks, we raise the minimum priority again. This way we will prioritize more important events whenever needed (and only lose events if really needed). This would allow us to have a very big queue in very big hosts and reduce amount of events being lost considerably (128GB ram host sizes could have ~16GB of cache, or allow user to specify the amount of cache, having in mind it will reduce lost events). That is why I mentioned, in our last talk, scalability studies rather than hand picking low hanging performance fruits, which is also valid but may not solve a core issue we currently have.
I think golang channels might do the trick, as long as we can measure how many buffers are being used at a particular moment (so we can decide on priorities and make If channels aren't good enough, we can always rely in regular async queues (perhaps with techniques to avoid GC to step into our feet). Implementation details are easy whenever we define the model. The only problem is that... currently, in open source tracee-ebpf, I don't have |
Beta Was this translation helpful? Give feedback.
-
That's not completely accurate, the files are being piped into a channel which is then passed to the
Considering the suggestion to use the go channels as the cache layer, that means we need to buffer those input channels. And if i'm not mistaken @mtcherni95 has a draft PR (#1353) which could be implemented to work based on the amount of objects in the channel buffers. |
Beta Was this translation helpful? Give feedback.
-
I think I wasn't clear in that sentence. What I meant is that, differently than CNDR, which uses buffered channel for the pipeline of
So, now, we only need to know size of this buffer depending on host numbers AND average throughput in worst case scenario with all rules being loaded. This can tell us how big this buffer should be. In a 2nd step we can implement the dynamic priority changes to eBPF events. |
Beta Was this translation helpful? Give feedback.
-
If I understand correctly, practically this means that even if we buffered the input channel to the |
Beta Was this translation helpful? Give feedback.
-
Yep, to document what we've just discussed (on Teams). That is exactly it. In open-source version I'll have to have a "generic" async buffer in the pipeline (to hold the events to be sent to stdout while tracee-ebpf stdin doesn't consume it). When we merge |
Beta Was this translation helpful? Give feedback.
-
Performance Observations (env: 5.15 kernel, 4 cpus, 4gb)
@danielpacak and I investigated some flaky smoke tests today. Smoke tests started being VERY flaky after we caused docker daemon to download and instantiate a container image, right before the test execution, instead of pulling the image first and then initializing the container.
What happened was that the workload of the docker daemon, in the docker hub node, was enough to make a very simple test (TRC-2) to fail significantly. I'm enumerating reproducers in multiple steps, here, isolating the variables:
The test is the following:
tracee-ebpfwhile true; do ps -ef; donein a 4 cpu boxstrace lsmultiple times (wait a bit so pipeline gets full)The tests:
Executed: 5
stracecommandsGot: 2 detections
Executed: 5
stracecommandsGot: 4 detections
ptraceevent. Keep the same pallalel workload (while true; do ps -ef; done).Executed: 10
stracecommandsGot: 10 detections
Some conclusions:
When loading all existing signatures (14), not much if we consider CNDR, we reduce the detection rate considerably (way more that if we select multiple events to be probed for a single signature). At least if that test is done in a small environment (4 cpus, 4 gb)
The fact that tracee-ebpf output is not consumed fast enough makes it to hold the pipeline, because the nature of the OS I/O handling, and then tracee-ebpf cannot consume the perf buffer fast enough. This makes ebpf programs to overwrite perf buffer events that were not consumed, causing event loss.
I do know that:
I'm wondering if we don't also need some sort of event caching in between the tracee-ebpf and tracee-rules (not relying only in the channel buffering only, when both are part of the same proccess, as it may exhaust very fast).
Don't we need something very fast to remove perf buffer pressure and allow tracee-rules to be slower if it needs to ? Eventually this cache would be drained during workload relief.
Beta Was this translation helpful? Give feedback.
All reactions