So, like with a lot of things it depends on your risk tolerance. If you have Android and don't take steps to keep Google from your location, the only extra thing your giving Google with Fitbit is your heart rate, activity level, weight, and menstrual cycle (if applicable). They'll (probably) sell this information in bulk to advertisers so they can link your advertisement ID to ads relevant to you, like ads for stationary bikes or whatever.
When faced with this problem what I said to myself was, "I can't selfhost and build a fitness tracker. So some company is going to get my info. I'll break it up as much as I can." I use Garmin.
Garmin watches are a quality product. I'm fond of the ones with eink displays so I don't have to charge the but once a month. Garmin has my health info. Proton has my emails. Google has my calendar events (Google calendar has no reasonable replacement imo.)
I know you didn't ask for other product recommendations. But all of this is a long winded way of saying the danger is letting one company have all your information.
Companies and governments will always know something about you, unless you live on the the moon. The important thing is keeping track of who knows what and if one company begins to know an unreasonable amount (looking at you Google), then start cutting off that company.
Lastly, consider your political climate. If I'm a woman in Texas I'm likely not going to let any company know when my last period was.