You will find your lack of fight against oppressers just more readily allows them to oppress.

Borders aren't a colonial invention, your reference is to the idea that common people are subject to borders. Lords had their realms, and even with that common people had security checks. Let us not forget the military checkpoints that Guan Yu was stopped at as he fled Cao Cao. Historical accuracy of the account aside; it shows military checkpoints in traveling roads.

If you want to say Rome was colonialist then I'd agree but you're going back three thousand years and refusing to acknowledge a system to prevent military incursions.

Regular people would get their wares checked and weapons confiscated when entering cities. Sometimes their wares would be held until they leave or they'd be denied entry.

The biggest change which you seem confused on is the scale of the states involved. Going from city state to country is more of a logistical hurdle we overcame than a colonialist expansion.