It really depends on what you want to accomplish, your priorities, the amount of time and effort you are willing/able to put into it, and your risk appetite (not just privacy but also availability of your mail server).

It is for sure one of the more challenging services to self-host, and IMO doesn’t offer a huge improvement over a hosted solution with your own domain from an actual security and privacy standards point since email is inherently insecure and non-privacy protecting without adding additional not-always-standard layers on top like PGP/GPG, SMIME, one-time passcode escrow systems, etc. that all have their own huge trade offs.

Your self-hosted server will have downtime as well, some planned but also some unplanned. If your server is down, it can’t accept or send mail obviously which can be an issue (many services will try to deliver again after a back off period, but won’t try forever). Enterprises work around this with load balanced servers and running different services on fault tolerant infrastructure. That increases complexity quickly though and isn't what most self hosters do AFAIK.