I agree with you on some points here. The problem is that these crawlers are hostile to the point of DDOSing sites

So the problem is not that someone archives your public accessible data, the problem is that in doing so either breaks your site or makes you pay for the excess traffic.

I think the web is now broken beyond repair. The commercialisation killed it and the tech monopolies are all that's left.

So I think small invite only fully encrypted enclaves are all that is left, until someone comes up with a "new Internet", that can resist the " Techbros", but for now I don't see that.

Also I don't see the Fediverse as a solution, it's just under the radar for now, but if it gets bigger it will be coöpted and sunk.