BPF Design Decisions

[rtkaratekid]

I've been working with BPF for a while, both writing and reading it. tracee.bpf.c is some of the best BPF code I've ever seen! I just wanted to express that to start out :)

I do have some questions about the design decision on the kernel side though, some of the paradigms are a bit unfamiliar to me in the context of BPF programming.

I noticed in tracepoint__raw_syscalls__sys_enter that the bufs and bufs_offset maps are used to aggregate the "context" (not the ctx from BPF but the context of the syscall) and then to aggregate the types, tags (to correlate an argument name in userspace I believe), and arg values. But then I kind of get lost around here

    if (event_chosen(RAW_SYS_ENTER)) {
        buf_t *submit_p = get_buf(SUBMIT_BUF_IDX);
        if (submit_p == NULL) { return 0; }

        set_buf_off(SUBMIT_BUF_IDX, sizeof(context_t));

        context_t context = init_and_save_context(ctx, submit_p, RAW_SYS_ENTER, 1, 0);

        u64 *tags = bpf_map_lookup_elem(&params_names_map, &context.eventid);
        if (!tags) {
            return -1;
        }

        save_to_submit_buf(submit_p, (void*)&id, sizeof(int), INT_T, DEC_ARG(0, *tags));
        events_perf_submit(ctx);
    }

So I read this as:

If we want to trace this event, get the buffer index of zero, put the context data into the buffer, then put... all the tags into the buffer? And then submit the data, without arguments, to the event perf buffer?

Those last two bits don't make sense to me. Why submit to the perf buffer prior to hitting the sys_exit code and aggregating the args? I looked at the userspace code which seems to suggest that you expect to get args each time you get a context...?

Then you hit the sys_enter_tails which basically just has special cases for handling exeve(at). That code I do understand because it's pretty similar to BCC's examples and similar to what I've written in the past as well.

Next is the sys_exit code, and this is where I'd expect to see the only perf buffer submission. We get the same pattern as what I listed above, but we build up another context and save the return value to it instead of 0, we save it to the buf map, then we submit to perf, again without aggregating the args?
So at this point I'm a little confused and thought that the sys_exit_tails would be where the arg magic happened, but it looks like it's actually never used since it's commented out in the go code.

So I decided to take a different approach and looked at some of the helper functions and came across trace_ret_generic which seems to add the args to the submit buffer and then submit the whole thing to perf.
I guess my questions are:

1. Why does it look like there are potentially three perf buffer submissions per syscall event traced?
2. With the buffer used to save and aggregate data, are there no issues with data races or conflicts? I haven't generated any but it feels like that's a possibility.
3. There seems to be a bunch of code that doesn't get used at all (like the vfs programs, security_bprm_check, security_file_open). I don't want to burden anyone with a full explanation, but am I just missing those getting tail called somehow? Following tail calls is pretty tricky when reviewing BPF source code.

Again, just wanted to compliment on the high-quality code! I've reviewed a bunch of the bigger BPF applications out there (and a bunch of smaller ones too) and this has been, by far, my favorite to learn about. It's really opened my eyes about what's possible with BPF.

[yanivagman]

Hi @rtkaratekid ,
Many thanks for the compliments! Knowing that people find your code helpful is very satisfying.

You got most of the things right, and I think that what you are missing is that Tracee (more specifically tracee-ebpf) is capable of showing not only system calls events. If you are using the latest docker image (v0.5.0), you can list all of the events that are supported by Tracee ebpf using the -l flag:
docker run -it --rm aquasec/tracee:latest trace -l
Note: In previous versions of Tracee, there was no need to provide the trace command, but this is now necessary as tracee-ebpf became just one of the components of Tracee (the other major component is tracee-rules)

By listing the events, you will see events like security_bprm_check and security_file_open. These are actually bpf programs by themselves, which are not called by any tail call youv'e seen in the sys_enter code (but are attached using kprobes, as you can see in the end of the big EventsIDToEvent table in the consts.go file). You can actually choose just one of these events, which will load the relevant bpf program youv'e seen in the c code by, for example:
docker run -it --name tracee --rm --privileged --pid=host -v /lib/modules/:/lib/modules/:ro -v /usr/src:/usr/src:ro -v /tmp/tracee:/tmp/tracee aquasec/tracee:latest trace -t e=security_file_open.

Getting back to your first question, note that when you list the events, there are also two events named sys_enter and sys_exit. You can use tracee to trace syscall entry and exit without any arguments by using these events (this "raw_syscalls" data can be used, for example, by machine learning algorithms that are based on syscall sequences and don't care about the arguemtns. Another use case is to check the time spent in each syscall by taking the diff between the entry and the exit). By choosing these events, the code that will trigger these events is exactly the code in you mentioned above:

    if (event_chosen(RAW_SYS_ENTER)) {
        buf_t *submit_p = get_buf(SUBMIT_BUF_IDX);
        if (submit_p == NULL) { return 0; }

        set_buf_off(SUBMIT_BUF_IDX, sizeof(context_t));

        context_t context = init_and_save_context(ctx, submit_p, RAW_SYS_ENTER, 1, 0);

        u64 *tags = bpf_map_lookup_elem(&params_names_map, &context.eventid);
        if (!tags) {
            return -1;
        }

        save_to_submit_buf(submit_p, (void*)&id, sizeof(int), INT_T, DEC_ARG(0, *tags));
        events_perf_submit(ctx);
    }

Now you can understand why there are three perf buffer submissions for each syscalls. Usually, when sys_enter and sys_exit events are not chosen (this is the default behavior) - two of these submissions will never happen!

Now for your last (second) question. Note that bufs and bufs_off are percpu maps. This means that each cpu gets its own copy of the buffer, and no inter-cpu race conditions can happen. Also, we aggregate the data on the exit from the syscall (in the case of syscall events), and our tracing programs can be preempted before they finish, so no race conditions should occur.

Hope that this answers your questions. You might also find the cmdline help usefull to explore other features of Tracee (like capturing file writes and some other stuff)

[rtkaratekid]

@yanivagman thank you so much for the great answer!

By listing the events, you will see events like security_bprm_check and security_file_open. These are actually bpf programs by themselves, which are not called by any tail call youv'e seen in the sys_enter code (but are attached using kprobes, as you can see in the end of the big EventsIDToEvent table in the consts.go file). You can actually choose just one of these events, which will load the relevant bpf program youv'e seen in the c code by, for example:
docker run -it --name tracee --rm --privileged --pid=host -v /lib/modules/:/lib/modules/:ro -v /usr/src:/usr/src:ro -v /tmp/tracee:/tmp/tracee aquasec/tracee:latest trace -t e=security_file_open.

I'm facepalming a bit here because I generally neglected reading the userspace code aside from the parts that deal with libbpf. I'm sure if I had just taken the time to better explore the cli options or the userspace code I would have seen this, sorry for the lack of effort!

You can use tracee to trace syscall entry and exit without any arguments by using these events (this "raw_syscalls" data can be used, for example, by machine learning algorithms that are based on syscall sequences and don't care about the arguemtns. Another use case is to check the time spent in each syscall by taking the diff between the entry and the exit)... Usually, when sys_enter and sys_exit events are not chosen (this is the default behavior) - two of these submissions will never happen!

Ah that makes so much more sense to me now! For some reason I assumed that all the code was meant to gather data and the arguments, but knowing this is just raw data collection makes the code so so much clearer for me.

Note that bufs and bufs_off are percpu maps. This means that each cpu gets its own copy of the buffer, and no inter-cpu race conditions can happen. Also, we aggregate the data on the exit from the syscall (in the case of syscall events), and our tracing programs can be preempted before they finish, so no race conditions should occur.

This is a great example of a use of these maps then, I just haven't found a use case for very many of the BPF map types but perhaps its time for me to rethink that and go back to fundamentals to review map options.

Thanks again, that answer was extremely helpful!