Of course the things you're doing in the present could cause large changes in the future. The problem is those changes are largely random, and it's often very unclear whether you'll be making changes for better or worse or just different.
The butterfly effect worry of traveling to the past isn't about you making intentional changes -- it's that any change could propagate through the chaotic system and have unexpected, seemingly random effects. It might change the future for the better, or for the worse, or just make things weird. But the crucial point is that the present you're used to and consider 'normal' -- your home -- might be irrevocably changed, perhaps ruined.
When you're just living life in the present, though, you're not worried about the possibility of changing the future because the future isn't yet set (in your perspective). You're not worried about what the future might gain or lose, because from your perspective in the present, the future doesn't have those things yet, so it can't gain or lose them.
And then there's the whole definition of 'changes'. Change as opposed to what? Every different possible choice or action might end up having far-reaching consequences in the future, even when you choose to do nothing. Maybe because you chose to stay at home and do nothing, you avoided a deadly car accident that would have otherwise happened and would otherwise have killed a guy. And then maybe that guy goes on to cure cancer ... or maybe that guy goes on to be a brutal dictator of a post-collapse nation. In either case, your choice to stay at home and do nothing has spared his life and greatly changed the future. And you'll never have any idea that you had any possible influence on whether that happened or not. You won't know that you changed anything, because it's the only reality you know and the only timeline you know.