Is there a reason why you want prowlarr and sab behind a VPN in the first place? If you just turn on TLS in sabnzbd you're not going to run into any issues and it's probably a better idea to search with prowlarr through your home address depending on what trackers you're using. I don't have either of these behind a VPN for my setup personally.
Gluetun just needs to share a docker network with traefik and not use network_mode. Do you have a default network in this compose file? If not you should add the same network name to every container and test to see if you can reach prowlarr with docker exec traefik wget -O - http://gluetun:9696.
Gluetun with protonvpn disconnects constantly for me and requires dependent containers to restart when it reconnects (https://github.com/qdm12/gluetun/issues/641) so make sure if you're testing stuff they're all restarted together
Nothing. I have all devices using tailscale DNS and I refer to things in my network by their host name directly.