This is exactly why this will never happen. Same reason the US only annexed the sparsely populated portions of Northwest Mexico after the war when they could have conquered the whole country. But gaining thousands of non-white citizens is too odious for white-supremacists to stomach.