Not completely impossible. You could have a default deny firewall, have your DNS resolver trigger an update to allow outgoing connections to the resolved IPs, and possibly also require connections pass though an SNI-sniffing proxy that only allows domains that your DNS resolver has allowed. Essentially by default you'd be blocking all custom protocols, and you'd only allow what looks like well-behaved TLS web traffic to allowed domains to flow.
Bad traffic could flow to a "good" domain, and then you need to decide whether that domain is actually "good".
couldn't they just hide their ad endpoints behind the proxy that serves their site? I can think of multiple ways to do this that aren't very difficult. I have had to implement something in my work to get past certain adblocking behavior that was going by domain
Sure, but now you've at least made them use a more expensive L7 proxy to do it, and you can decide to block malicious actors like that entirely (blocking the "good" domain).
Yes, you can do it with an L7 proxy. You've been able to do that all along though, so I suppose there are reasons why surveillance networks prefer to not proxy through the websites that host their scripts. That has nothing to do with DoH to subvert network security monitors though.
Bad traffic could flow to a "good" domain, and then you need to decide whether that domain is actually "good".