The insane part is that since the website is powered by WordPress, the scrapers could access all the posts in a single JSON file.
I was also exasperated by the fucking scrapers reading the same fucking page 20 times a second on posts that didn't get new content from a decade ago so I migrated my blog to Hugo and I was completely shocked to discover that by default every WordPress blog comes with an unauthenticated api that allows literally everyone to get the whole blog content in JSON files. Why the fuck are you wasting my server power to scrape the HTML if you can get a easy JSON??????? Take that fucking JSON and subscribe to the RSS for getting the next post that will be published the next decade. If you refresh that fucking URL 1000 times a day you will get the same fucking stuff, not a new magical article
I mitigated the issue with Wordfence by setting a rule like "more than 2 pages requested within a second = ip banned for a year"
Now, why would WordPress include an unauthenticated api that allows everyone to do a full unauthorized copy of the site in literally seconds is beyond me. There's no valid reason to have it public without any authentication. That API shit doesn't make sense, why by default a website should accept user signups from bots via API