We're seeing this at work too - our public git frontend is constantly getting scraped as well as our self hosted issue tracker. We had to spend days working on fail2ban and other kinds of tools to mitigate all the traffic that's adding tons of load to our instances, which otherwise would easily be able to handle the handful of employees who actually use these systems.

[–] Botzo@lemmy.world 23 points 2 weeks ago (1 children)

Looks like this is the written version of the video:

https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/

[–] AbnormalHumanBeing@lemmy.abnormalbeings.space 11 points 2 weeks ago

Ah, yeah, I keep being scatterbrained and impulsive when crossposting, and forgetting he sadly doesn't even link direct links in the video description, instead of just to the website itself. Will add the link to the lemmy post bodies at least.

[–] floofloof@lemmy.ca 14 points 2 weeks ago

There are tarpits like Nepenthes but they use up your CPU resources and I imagine it would be pretty easy to update a scraper to recognize these generated pages, since they're all structurally similar.

[–] nomugisan@lemmy.dbzer0.com 11 points 2 weeks ago

I'm no Sysadmin but to me it sounds like we need a botnet/scraper resistant application layer protocol to replace HTTPS.

[–] haverholm@kbin.earth 8 points 2 weeks ago

So basically what Drew DeVault wrote about the other day I guess?