There may be some plan related limitations if you’re swapping from a smart phone to a “dumb” phone. That’s something that’s best to speak with an AT&T rep directly about.
As for the SIM card, yes you can walk into a store and request a physical SIM. There should not be any charge for that as you’re already a paying customer. Do not let anyone else fool you otherwise. Source: I used to work with cellular devices and am familiar with most AT&T policies.
Also, avoid AT&T franchises. When you walk in, simply ask if this is a corporate location. If they tell you no, I would try elsewhere.