Degraded performance in virtualized environment

[jfernandez]

I normally use virtme-ng with a Thinkpad X1 Carbon running Ubuntu 22.10 and 6.5 kernel. That setup has no issues AFAIK. I just got a new laptop (Lenovo Slim Pro 9i), which is running Ubuntu 23.10 and the 6.7.9 kernel (to fix a keyboard issue). I've verified that virtualization is enabled in the BIOS. This new laptop has a noticeable performance degradation problem inside the virtualized environment.

I use vng --rwdir . to be able to compile selftests inside the VM. With the new laptop, running LLVM=1 make -j20 is incredibly slow, and it doesn't look like its using all my vCPUs. Other issues I've encountered:

• Trying to autocomplete a path by hitting tab makes the whole VM unresponsive and I need to kill the terminal.
• does ls -al inside the bpf selftests root dir also makes the VM unresponsive.

[arighi]

Trying to guess here, maybe a filesystem performance issue? Are you using 9p for your new rootfs instead of virtiofs? What does df -hT / say?

[jfernandez]

Hm, I tried without --rwdir . and the issue went away. And it came back after I used the flag.

The only reason I started using --rwdir is because in my other laptop, the Thinkpad, I would get write denied errors without it.

[jfernandez]

It's happening in several kernels, including going back to 6.5. It seems like its the --rwdir flag

[arighi]

Ok, so --rwdir uses 9pfs that has much slower performance than virtiofs. If you use --rwdir in your entire $HOME, for example, and you do a lot of filesystem operations, performance will be degraded a lot.

Theoretically if you don't care about saving the results of your tests somewhere you can avoid using --rwdir and everything should work.

What kind of test are you running that gives you permission denied errors? I'd like to try to reproduce the problem on my side (like if you have a small reproducer it'd be great).

On a more general perspective, we should use virtiofs also for --rwdir mounts, but the problem is that each virtiofs mount point requires a separate virtiofsd running on the host, that means we need to start multiple instances of virtiofsd for each --rwdir. Not terrible, but also not very elegant... And considering that --rwdir has been always used rarely, optimizing its performance has never been a priority before.

I can take a look and try to create a PR to switch to virtiofs for --rwdir mounts, but first I'd like to understand better why you're getting permission denied errors in your test. :)

[jfernandez]

@arighi I'm no longer getting the write denied errors when I don't use --rwdir in my old Thinkpad. I tried to reproduce it, but it's not happening again.

I'm fine with not using --rwdir going forward. The only reason I thought I needed it was to compile selftests inside the VM. I would hold off on prioritizing any work due to this for now. I will follow up if the write denied error happens again.

[arighi]

ok thanks for the update! Improving --rwdir performance is still in my TODO list, but it's good to know that you don't necessarily need it right now for your workflow. :)