I use Pop and have a Gnome extension that helps with this. I’m not at home now and I can’t remember the name but I can report back later.
Doing a quick search on this gave me “Application Volume Mixer” but I don’t think this is the exact extension I have.
Okay the extension I have is called “Sound Input & Output Device Chooser.” I think that if you install this and the Other extension above your problem will be solved.