Yes and no. The one the Nazis used was / is also a symbol used in Asia. (And really, everywhere. It's a fairly basic pattern.) The Nazis used it because of their obsession with Aryans. Sometimes it's at an angle, sometimes it's not.
Generally in the west, unless it's on a statue of budda, any swastika-ish symbol since the ~1930s is going to be a reference to Nazis though.
I now deeply want to hear the call to prayer sung in English in a heavy Minnesotan accent. Any accent really. LA, Texan, Georgian (state, not country; you know what, either).