I'm not suggesting everyone must go vegan, but I guarantee there are Buddhists in your community in similar economic positions that are vegan or vegetarian. It sounds like it's more a mental bandwidth thing than a requirement. It's objectively cheaper to eat vegan if you subsist on rice and lentils, like much of the most impoverished do. Again, I'm not suggesting you need to do anything, but the idea that going vegetarian or vegan is an elitist or privileged thing is objectively false, economically. You could absolutely go vegan for cheaper than you are eating now, if you were willing to restrict your diet. It's not the right decision for everyone, but don't make it about money or privilege.

Not going vegan or vegetarian because you have more important things to worry about than a diet change is a totally valid reason. Don't make excuses or punch left because your feelings are hurt. You can choose to not go vegan or vegetarian for your own reasons without shaming those that make the difficult choice to do so.