The short version: I have a very high system load (load is 17 with 4 CPUs), and cannot pinpoint the reason, on a 22.04.3 system with kernel 6.5.0-15-generic. But it appears to be related to the number of currently mounted Samba/CIFS shares from other devices on my LAN. I'd really like some troubleshooting help here, or a workaround if this is a known issue that I haven't been able to find by googling.I can't find the bottleneck, no system monitoring tool or dmesg/journalctl output shows me any sign of why the load is that high. Any ideas?
More background and longer version: I have a couple of Ubuntu systems, from a desktop to Raspberry Pis and other single-board computers, running Ubuntu 22.04 LTS or 20.04 LTS. I have not touched any configuration that I suspect can be related for several months, but I recently (in January 2024) noticed a ridiculously high system load without any obvious cause on one of them. Problem persisted after reboot. (These systems all run unattended-upgrades, so they are more or less up-to-date, but it often goes weeks between reboots so the kernel version might be different.)
I also noticed a very high utilization (> 16) on my desktop (8-core/16-thread AMD CPU) once after a Samba share crashed, but that problem did go away after a reboot, haven't been able to reproduce it.
System info (where I did reproduce the error): Ubuntu 22.04.3 LTS, 4 cores, Intel Celeron J4105, 8 GB RAM. uname -a: 6.5.0-15-generic #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
Tests and test results: I stopped all high-CPU or disk-intensive processes, and tested different number of mounted Samba shares. In all cases, iostat reported avg-cpu %idle to be > 97 (if I didn't run iotop, which when I did increased %system to 2-3 and reduced %idle to about 90). iostat also reported disk %util< 1 and practically zero transactions per second, and iotop didn't show anything either, so the load does not seem to be disk-related. htop showed no process (except iotop) to use > 5% CPU.The Samba shares were mounted, but not used: lsof did not list any files under /media or /mnt (where these mountpoints are).
To me, the load average should be very close to zero here. But it isn't.
- Zero (0) Samba shares mounted: Load was about 0.2. So practically zero.
- Two (2) Samba shares mounted, one share per external Ubuntu system on my LAN: Load was about 4.
- Four (4) Samba shares mounted, to two separate systems on my LAN: Load was about 8.
- Ten (10) Samba shares mounted, to five different systems both on LAN and via VPN: Load was about 17.
Counterexample 1: A Raspberry Pi 4, running Ubuntu 20.04.06 LTS (i.e., not 22.04) with uname -a showing "5.4.0-1100-raspi #112-Ubuntu SMP PREEMPT Fri Nov 24 15:35:17 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux" and 20 active Samba shares mounted had a load of about 0.2. Again, practically zero.
Counterexample 2: My desktop, running Ubuntu 22.04.3 LTS (same as system above, but an AMD CPU) with uname -a showing "6.2.0-39-generic #40~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 16 10:53:04 UTC 2 x86_64 x86_64 x86_64 GNU/Linux" and 16 active Samba shares mounted had a load of about 0.3. Again, practically zero.
Note: I just noticed that I don't have the same kernel version on both x86_64 systems. But looking in /boot, both of them got 6.5.0-15 installed on the 12th of January. So I guess I could reboot my desktop and select the 6.5.0-15 kernel, and see if the problem appears on the desktop too.
Other things I've tested: By googling, some people seemingly got better Samba performance by adding "vers=3.0" to the cifs lines in /etc/fstab. I tried both with and without, but did not see any difference. A reboot did not help.
Suggestions on the way forward are welcome...
Edit: I did reboot my desktop and chose the 6.5.0-15-generic kernel in Grub (uname -a: "6.5.0-15-generic #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux"), and had 16 Samba shares mounted: The system load jumped up to 27 - But the system seems as responsive as ever.
My conclusion is that the 6.5.0-15-generic kernel reports the load incorrectly when Samba shares are mounted.
But what is the next step? How to mitigate it, except pinning the older kernel? And where to report the bug?