Thanks. That seems to be a similar, but slightly different error. I think the below may apply though.
I believe I've tracked down more of my issue, but fixing it is going to be a hassle:
When cloudflare proxying is enabled, there are 3 DNS records involved; A record with cloudflares ipv4, AAAA record with cloudflares IPV6, and the key to this puzzle: an HTTPS record with cloudflares ech/https config.
With pihole I can set DNS records for A/AAAA, but I have no way of blocking/setting the HTTPS record so it gets through from cloudflare.
The LAN A/AAAA records don't match the HTTPS record from cloudflare, so browsers freak out.
Once I disabled cloudflares proxying, I no longer get HTTPS records returned and all works as intended.
I'll either have to keep cloudflare proxying disabled, or switch pihole out for a more comprehensive DNS solution so I can set/block HTTPS records :(
Thank you @bobslaede@feddit.dk for pointing me in the right direction.
I think I've found the problem:
It seems my issue is pihole being unable to block/modify dns requests for HTTPS records, which don't match the LAN IPs pihole handed out in A/AAAA records.
I've disabled cloudflare proxying so they don't have HTTPS records to serve, but I'll have to replace pihole with a better lan DNS solution if I want to turn that back on.