Not likely worth it. Primary reason is that the large federated email services are skeptic also of email from services such as your proposed self hosting solution and may simply not deliver the mail you send. This is to mitigate against spammers setting up a bespoke servers.
There are a bunch of other things that could go wrong if you don’t set everything up perfectly, but even if you do, this would be a big problem.
Better off using a custom domain with a big provider. Fewer headaches. I like Fastmail, but many others are great too.
The criticism I often hear for this is from drivers. My question is always “Would it be better for you if I got off and walked and you had to wait 2-3x as long for me to cross?”