It's a lack of nuance. Higher rates of population growth can be good, if pressure points like housing are planned for with zoning and permitting systems that promote densification in popular locations. The badness is neither the additional people nor the housing regulations individually, but instead is that they don't match.

Also, there's a lot of racism in the mix. The people with legitimate concerns about growth planning (or the lack thereof) end up mixed in with the people who are horrified at the idea of their racial group becoming a minority of the country's population.