You could make that same argument to the countries neighboring Nazi Germany.
We have examples in history of what happens to unarmed people when fascists take over and few fight back, and it does nothing to quell the fascist's efforts or 'fears'. We also have examples of armed people fighting back, like the leftists in the Spanish Civil war. Their defeat was not a given, and they made the fascists work bloody hard for it. The alternative would've been the leftists having to blend in or be disappeared/killed, or they could've left everything and fled. The less you fight them, the stronger they become, until they become too big to run from or ignore.
Both conflicts are horrific, but what was their alternative? We saw what happened in Germany when few fought back, and that was just as horrific an outcome, if not more so (6 million Jews killed vs 300 thousand on the left side in the Spanish civil war, though estimates vary).
Tens of thousands died under Mussolini in labor camps and via execution, and the same would've happened under Franco in Spain (and eventually did, post civil war)
To be clear, I'm not advocating that any country rush to armed conflict, but history seems to indicate that it's better to be capable of defending yourself vs. not having the option at all.
If you have examples of pacifism being effective against fascism, I'm quite open to having my mind changed. In fact, I would prefer if that were the more effective option, if evidence supports it.