Since you're talking about Germany: Tofu has been unfairly demonized here (maybe because it can serve as a meat substitute).
It is a great source of nutrients and protein. It can be prepared in many different ways. It is environmentally friendly, can be conveniently stored and has a relatively long shelf life.
I've seen a lot of hate towards vegans online, admittedly being someone who partook in that several years ago myself to a small degree.
Then you probably understand that the hate is primarily a coping mechanism because vegans propose arguments that question their lifestyle. It's a "them" problem, not a "you" problem.
If you value consistency, don't let a few jerks push you into a behavior that isn't true to yourself. Don't come up with fake reasons for why you neglect to partake in rituals that don't respect the rights of animals.
State your real reasons. This is also better for the animals. If someone then wants to start a discussion, you can still politely refuse.
Although I agree with the rest of the comment, I would suggest a less brutal activity, such as a hiking or bird watching.
These also require less equipment.
Now imagine there's an outdoor barbecue and you're a pig and the only person speaking up for you is disregarded by everybody else as a dysfunctional nutter.
Most of the experience, including your examples, is not specific to Germany. The notion that tofu is not for "real men" can be seen in other meat-heavy societies. Just like the idea that tofu is not a filling meal. In pop culture, it is often associated with weak traits (e.g. "soy boy"), and described as tasting bad or bland in general.
In Germany, a special permit was required until the 1990s to produce certain types of tofu. If you offer people something with tofu, many still turn up their noses or laugh because they think you're joking. I don't notice this as much with any other widely available food.
There is also far too much discussion about whether something can be called tofu sausage, tofu burger or tofu butter. But I think here we've passed the low point and common sense is slowly gaining ground.
Sometimes you still come across the false claims that tofu raises estrogen levels and that the rainforest is cut down to produce it.