Some woman shop for/wear "men's" clothes, either because they shop for the men in their life, or for themselves because the standards are more sensible (even if not perfect) compared to women's sizing. In other situations, we wear "men's" cut clothes because it's the default - like when a workplace gives everyone a free T-shirt. 9 times out of 10, it's probably a cut designed for men - even if the workplace has a majority of women (as was the case when I worked in a nursing home.)

At least for pants, a lot of men's pants sizes usually go off a band + length measurement, which is a ratio that women's clothes don't offer at all. T-shirts can be bad either way, but I once grabbed two ("women's") shirts off the same rack in a store and both fit me perfectly - one was Small, the other was Extra Large. I've never seen that bad of a difference when trying on "men's" clothes, and that's part of why I prefer to buy from the men's section. It's more sensible.

So yeah, vanity sizing hurts everyone. But unless you do shop for both men's and women's clothes, it's hard to appreciate just how awful vanity sizing is for women in particular.