Can't get trace on EKS with error "linux/types.h' file not found"

[mhausenblas]

I have a vanilla EKS cluster and tried the first example here, that is:

$ kubectl trace run ip-XXX-XXX-XXX-XXX.eu-west-1.compute.internal \
                 -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); }"

… however nothing happens. When I then look at the logs, I get:

$ kubectl trace logs kubectl-trace-ccf26276-03ef-11eb-bb6a-4865ee1b6baf                                                                                                                                                                             if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
/bpftrace/include/asm_goto_workaround.h:14:10: fatal error: 'linux/types.h' file not found
exit status 1

Any idea what I am doing wrong?

[howardjohn]

I also see this with GKE v1.18.6-gke.4801

  nodeInfo:
    architecture: amd64
    bootID: 1d45c4af-5438-4d5e-a153-1036dc69a8cb
    containerRuntimeVersion: docker://19.3.9
    kernelVersion: 5.4.49+
    kubeProxyVersion: v1.18.6-gke.4801
    kubeletVersion: v1.18.6-gke.4801
    machineID: ec7958c51f94158d4815fc14ad9f4614
    operatingSystem: linux
    osImage: Container-Optimized OS from Google
    systemUUID: ec7958c5-1f94-158d-4815-fc14ad9f4614

[mhausenblas]

Thanks for confirming @howardjohn and wondering if anyone here had a chance to look into it?

[mhausenblas]

Is this project dead?

[leodido]

Can you give us the output of ls -la /lib/modules/$(uname -r) please?

I think you need to install the kernel-devel-* package on the node to get it working, BTW.

First attempt to try would be to use the --fetch-headers flag.

[mhausenblas]

Thanks @leodido for following up, clearly not dead ;)

Can you give us the output of ls -la /lib/modules/$(uname -r) please?

Will do ASAP, yes.

[mhausenblas]

[ec2-user@ip-192-168-19-137 ~]$ ls -la /lib/modules/$(uname -r)
total 1200
drwxr-xr-x  7 root root   4096 Jul 24 16:02 .
drwxr-xr-x  3 root root     43 Jul 24 16:02 ..
lrwxrwxrwx  1 root root     46 Jul 24 16:02 build -> /usr/src/kernels/4.14.186-146.268.amzn2.x86_64
drwxr-xr-x  5 root root     42 Jul 24 16:02 extra
drwxr-xr-x 12 root root    130 Jul 24 16:02 kernel
-rw-r--r--  1 root root 170617 Jul 24 16:02 modules.alias
-rw-r--r--  1 root root 180101 Jul 24 16:02 modules.alias.bin
-rw-r--r--  1 root root    292 Jul 14 18:19 modules.block
-rw-r--r--  1 root root   3041 Jul 14 18:19 modules.builtin
-rw-r--r--  1 root root   4927 Jul 24 16:02 modules.builtin.bin
-rw-r--r--  1 root root  83649 Jul 24 16:02 modules.dep
-rw-r--r--  1 root root 118851 Jul 24 16:02 modules.dep.bin
-rw-r--r--  1 root root    383 Jul 24 16:02 modules.devname
-rw-r--r--  1 root root     13 Jul 14 18:19 modules.drm
-rw-r--r--  1 root root     13 Jul 14 18:19 modules.modesetting
-rw-r--r--  1 root root    231 Jul 14 18:19 modules.networking
-rw-r--r--  1 root root  32866 Jul 14 18:19 modules.order
-rw-r--r--  1 root root    580 Jul 24 16:02 modules.softdep
-rw-r--r--  1 root root 264809 Jul 24 16:02 modules.symbols
-rw-r--r--  1 root root 322035 Jul 24 16:02 modules.symbols.bin
lrwxrwxrwx  1 root root      5 Jul 24 16:02 source -> build
drwxr-xr-x  2 root root      6 Jul 14 18:19 updates
drwxr-xr-x  3 root root     57 Jul 24 16:02 vdso
drwxr-xr-x  2 root root      6 Jul 14 18:19 weak-updates

[adelbertc]

As @leodido mentioned you may need to run with --fetch-headers. I got the same issue on GKE until I started using --fetch-headers.

[dalehamel]

GKE in M85 (1.18) should include BTF. I am going to be doing some work on bpftrace to ensure the image we use in kubectl trace can take advantage of this.

We also use GKE, which is why I made the --fetch-headers flag, since they don't have an easy way to get headers. However, once BTF is fully supported there, it will be much prefered.

[ghost]

So to get the tracing running properly you need to have linux-headers-$(uname -r) installed on debian based systems; this correct? I assume that's the dependency for all nodes? Anything else need to be installed on the nodes? Just headers?

[dalehamel]

Headers yes, or use the “—fetch-headers” flag (which will slow down starting the trace significantly, but ought to ensure correctness).

support for BTF and IKHEADERS should be coming soon for kernels that support it.

for now, what we do at shopify is use a daemonset with an init container to pre-install headers on all nodes, using the same script as what is done with “—fetch-headers”.