this post was submitted on 21 Jul 2026
62 points (100.0% liked)
Fuck AI
7768 readers
1759 users here now
"We did it, Patrick! We made a technological breakthrough!"
A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.
AI, in this case, refers to LLMs, GPT technology, and anything listed as "AI" meant to increase market valuations.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
glue traps are a good solution to badly-behaved scrapers. just list a path in your robots.txt that slowly produces random text and links, all of which go to the same sort of page. if they respect robots.txt, they're fine. if they don't, they get stuck ingesting nonsense.
not a catch-all solution but a good tool.
AFAIK it was a glue trap / tarpit that one guy I know tried. Sure, it did result in the scraper getting a lot of useless data. OTOH, they dedicated massive resources to trying to get those non-existent documents, so his website was effectively DDOSed as a result. That's why he's now blocking entire countries.
yeah slowing down access and using a cheap generation method is important there.
Even that isn't enough if they respond to that by just throwing more crawlers at your site.
yeah it's not the only thing you want on there of course. i run a tarpit, fail2ban, and anubis.
Yeah, at one point he posted that fail2ban was only able to ban about 20 IPs/second, but he was getting 200 requests per second, and the bans weren't having a meaningful impact on that (i.e. the requests were all coming from different IPs). From what I remember, his system load was over 1000.
oh wow. vps? they tend to live in busy ip blocks.
I think it was running on a VPS, but I'm not sure. As for the traffic, I think it was coming from all over. Amazon-owned netblocks, mysterious ones that were hard to trace, etc.