KDE's audio panel is not a switch for the general sound server on your machine, just for KDE. Each application running can still access and get a mixer for your sound devices unless you're using an app which is globally controlling certain settings.
Some extra info here on the inner workings and mixing. It sounds like you want your settings to be an all or nothing situation for all of your output devices, so maybe focus on syncing them up, or find an output profile manager or switcher like this one (it's a bit old though).