I have 0 apps allowed microphone access all the time.

There is no evidence that phones are snooping on people, but I would say even if unlikely it's a reasonable concern given what companies do get up to.

However it is more likely the ads were being served because of all the other data you're allowing Google to scrape from you all the time rather than the phone mic.

Rather than focusing on the microphone, look at the bigger picture of how your data is being pillaged by Google all the time.

For me, I switched away from Gmail, stopped using their search engine, use Firefox and not Chrome, and don't use their other services where possible. I have android on my phone and use Google maps and Google home. It's still a huge problem but I use that part of the ecosystem for convenience and no other. Similarly on PC I don't use Google for anything where I can avoid it, use Firefox containers to keep Google, Amazon, Microsoft, and Meta data as separate as possible, plus I use Linux and VPN as needed, and lots of privacy extensions in Firefox.

It's possible to minimise your data exposure to the big tech companies, but difficult to severe completely. You could go even further and switch from android to Graphene OS (I have seriously considered this).

I would go by the principle of compartmentalising your data as much as possible and limiting access to snooping eyes. The transition can be hard but once you've done it you get used to using disparate unnonnected services. Like I really don't need or benefit from my email data being connected to my data storage or my search engine; it's a false convenience that benefitted Google only.