Depends on where you live and what your building.

In some areas you need a full foundation on others it may not be necessary

Don't need a garage? Then you save money just having an exposed asphalt driveway

Cellar? Fireplace? Attic? Even a 1000 square foot home has a lot of variation in design and material requirements

If anything I'd say it's the labor that drives the price into the stratosphere. But even that varies county to county, even more so state to state.