I've never once pushed my diet on anyone or brought it up out of necessary contexts

And that's a good way to go about it. But unfortunately that's not the case for every vegetarian or vegan, and as for every demographic, the loud and confrontational ones are always the ones that get the most attention. And this is coming from an atheist that doesn't talk about religion unless directly asked about it, so I can sympathize.

The sad fact is that most people that are wary of vegans and vegetarians are like that because they've run into ones that are varying forms of asshole about their diets. Some of the antagonists throw insults, some will be dishonest, most will be some mix of the 2. Of course it's not fair that those bad impressions are being applied to you as an individual, but it's unfortunately also just human nature.

Just going through the other responses on here. Pretty much every other responder is trying to come up with some complicated Psychology 101 argument as to why people don't like vegans despite the answer being simple.

Hell, I've run into vegans criticizing keto (which is weird because you can be vegan/vegetarian on a keto diet), and their tactic was to throw out something like 14+ research papers and pretend that they all backed up their claims. I went through all of the papers, and 1/3 were completely irrelevant, 1/3 weren't even looking at proper keto (or even properly low-carb) diets, and the final 1/3 contraticted their points.

While this may have seemed like a tangent, it's relevant to my point: people are messy, and they'll sometimes be dishonest and/or contradictory, especially when they think they have the moral high ground. And people that are on the fence or disagree can see that dishonesty and it leaves a bad impression for the whole group.

To recap: just like with atheism, a lot of the issue is a small number of bad apples that give the group a bad name. All you can do is mind that fact and try to avoid being one of the bad apples yourself.