Software center: "Prepared update not found"

So I recently updated to Xubuntu 22.04. Now the software center claims to have some updates which "Requires Restart". However, when I click the "Restart and Update" button, there is a small popup claiming

Unable to install updates:Prepared update not found: /var/lib/PackageKit/prepared-update

Is that some kind of stale state? How can I solve this? It's pretty annoying that Software Center claims to have updates, but is not able to install them...

Here is a screenshot from software center: