while I was writing this comment I came across this: LinVAM which sounds like exactly what you are looking for. But, if that doesn't work out for you here's what I was originally writing:
Voice Attack may fit your needs.
BUT
- it's not Linux native.
- It's not free.
However, my research does suggest that it works in Linux via proton/wine, and so it may serve your needs since what you've described is basically exactly that software's whole purpose. It's popular for adding voice control to games by mapping voice commands to game controls.