You say drove them out but that's not true. The vast majority of residents stayed. It's not like all the Christians left, most of them stayed and paid the tax or converted. The so-called reconquest was of people that had been living there for hundreds of years and whose ancestors have been living there even longer.

Also important to remember that the visogoths were a very small ruling class. A very recent one too.