I'm running virtualization on Ubuntu 22.04lts host and when upgrading the kernel to 6.* we start to have issues with latency on the VM machine(qemu+libvirt). The moment I roll back to kernel 5.15 on the host the latency issues are gone. Even when I restart the VM while still on kernel 6.* as you can see on the graph below issues are gone but only for a short period. Tried installing the latest Ubuntu 23.10(kernel 6.5.0-25-generic) assuming the problem is between newer kernel 6+ and older libvirt inside Ubuntu 22.04lts, but the issue is the same. At the same time when VM has a problem with latency, host latency is good. Also, I created a VM with 1CPU and 1GB on the same host and it works good all the time, no latency issues at all. Tried NUMA CPU pinning, but the problem remains the same. Something is wrong with 6+ kernels, any help would be appreciated. All the sorts of hardware options have already been tried, 100% not hardware-related. I found that when latency issues start we have this on the vnet0 of the VM
TX errors 0 dropped 2956864 overruns 0 carrier 0 collisions 0
Host is HP ProLiant DL385 Gen10 Plus v2CPU: 2x AMD EPYC 7543 32-Core Processor with 256GB RAM.Network card is BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller
VM is 40CPU and 145GB RAM