wingiee

joined 1 month ago
sorted by: new top controversial old
Everyone knows all the apps on your phone in c/android@lemmy.world
Cloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives. in c/technology@lemmy.world
[–] wingiee@lemm.ee 20 points 1 month ago (2 children)

I assume a crawler which adheres to robots.txt