How can I solve the DNS leak when using systemd with DNSOverTLS

The problem:

I am connected to a foreign wifi on Kubuntu 22.04 and want to configure the DNS to 9.9.9.9 with DNSOverTLS system-wide. However, on dnsleaktest.com I still get a different DNS.

Here is what I did:

In NetworkManager for the affected wifi, I disabled IPv6 and set IPv4 to Automatic (Only addresses) and put an empty list as DNS server. The resulting configuration in terminal is:

$ nmcli deviceDEVICE        TYPE      STATE         CONNECTIONwlo1          wifi      connected     TP-LINK_58DEp2p-dev-wlo1  wifi-p2p  disconnected  --lo            loopback  unmanaged     --$ nmcli device show wlo1GENERAL.DEVICE:                         wlo1GENERAL.TYPE:                           wifiGENERAL.HWADDR:                         xx:xx:xx:xx:xx:xxGENERAL.MTU:                            1500GENERAL.STATE:                          100 (connected)GENERAL.CONNECTION:                     TP-LINK_58DEGENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/1IP4.ADDRESS[1]:                         192.168.0.112/24IP4.GATEWAY:                            192.168.0.1IP4.ROUTE[1]:                           dst = 192.168.0.0/24, nh = 0.0.0.0, mt = 600IP4.ROUTE[2]:                           dst = 0.0.0.0/0, nh = 192.168.0.1, mt = 600IP4.ROUTE[3]:                           dst = 169.254.0.0/16, nh = 0.0.0.0, mt = 1000IP6.GATEWAY:                            --

I configured /etc/systemd/resolved.conf as such (only the last few lines are relevant, but for completeness / error checking I provide the full file):

$ cat /etc/systemd/resolved.conf#  This file is part of systemd.##  systemd is free software; you can redistribute it and/or modify it under the#  terms of the GNU Lesser General Public License as published by the Free#  Software Foundation; either version 2.1 of the License, or (at your option)#  any later version.## Entries in this file show the compile time defaults. Local configuration# should be created by either modifying this file, or by creating "drop-ins" in# the resolved.conf.d/ subdirectory. The latter is generally recommended.# Defaults can be restored by simply deleting this file and all drop-ins.## Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config.## See resolved.conf(5) for details.[Resolve]# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com# Google:     8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google# Quad9:      9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net#DNS=#FallbackDNS=#Domains=#DNSSEC=no#DNSOverTLS=no#MulticastDNS=no#LLMNR=no#Cache=no-negative#CacheFromLocalhost=no#DNSStubListener=yes#DNSStubListenerExtra=#ReadEtcHosts=yes#ResolveUnicastSingleLabel=noDNS=9.9.9.9#dns.quad9.netFallbackDNS=9.9.9.9#dns.quad9.netDomains=~.DNSSEC=yesDNSOverTLS=yes

After sudo systemctl restart systemd-resolved and sudo systemctl restart NetworkManager, this is the resulting configuration:

$ resolvectl statusGlobal           Protocols: -LLMNR -mDNS +DNSOverTLS DNSSEC=yes/supported    resolv.conf mode: foreign  Current DNS Server: 9.9.9.9#dns.quad9.net         DNS Servers: 9.9.9.9#dns.quad9.netFallback DNS Servers: 9.9.9.9#dns.quad9.net          DNS Domain: ~.Link 2 (wlo1)Current Scopes: none     Protocols: -DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=yes/supported

In basic tests it seems to work:

$ resolvectl query archlinux.comarchlinux.com: 3.64.163.50                     -- link: wlo1-- Information acquired via protocol DNS in 1.1292s.-- Data is authenticated: no; Data was acquired via local or encrypted transport: yes-- Data from: cache network$ dig archlinux.com; <<>> DiG 9.18.18-0ubuntu0.22.04.2-Ubuntu <<>> archlinux.com;; global options: +cmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60948;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1;; OPT PSEUDOSECTION:; EDNS: version: 0, flags:; udp: 65494;; QUESTION SECTION:;archlinux.com.                 IN      A;; ANSWER SECTION:archlinux.com.          7200    IN      A       3.64.163.50;; Query time: 331 msec;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP);; WHEN: Sat Apr 06 09:30:08 CEST 2024;; MSG SIZE  rcvd: 58$ ss -tuna | grep :853tcp   TIME-WAIT 0      0           192.168.0.112:48420        9.9.9.9:853tcp   ESTAB     0      0           192.168.0.112:49032        9.9.9.9:853

The remaining question though is: Why do I get a different DNS on dnsleaktest.com?

Using Firefox with "DNS over HTTPS" set to Off, on dnsleaktest.com I get:

In the same browser on https://on.quad9.net I get: YES, You ARE using quad9

Some things I tried:

• sudo dpkg-reconfigure resolvconf (/etc/resolv.conf only contains nameserver 127.0.0.53)
• sudo resolvectl flush-caches
• sudo resolvconf -u
• Put 9.9.9.9 only in the IPv4 DNS server list in NetworkManager
• Added dns=none to the [main] sections of /etc/NetworkManager/NetworkManager.conf and /etc/NetworkManager/conf.d/90-dns-none.conf
• sudo systemctl restart systemd-resolved
• sudo systemctl restart NetworkManager
• reboot
• Followed this guide which uses a simpler configuration: https://www.ubuntubuzz.com/2022/04/enable-browsing-with-dns-over-tls-dot-on-ubuntu-made-easy.html

Nothing yet helped. Can you help me solving this?