The problem is you lose the ability to encounter new communities by browsing "All". And since I'm the only user on my instance I will never discover any communities that I don't manually add or find on third-party websites not to mention that process of using those websites is cumbersome and tedious...
I added the top few thousand communities and the server runs absolutely fine. :)
https://github.com/jheidecker/lemmony
This will take care of the other commenters point about having to manually subscribe to everything. I just had it pull the top 500 communities from the top two instances and I'm fine...