Yes, though you would have to remember more numbers if you’re not making gendered sizes, things like the diameter of the arm hole and the distance between the neck and shoulder, but it’s not like there’s no way to do it. It’s just easier for clothing companies to gender clothes and most people don’t care enough to do anything about it.
People’s clothes would probably fit a lot better though.