Open Source

48527 readers

225 users here now

All about open source! Feel free to ask questions, and share news, and interesting stuff!

Useful Links

Rules

Posts must be relevant to the open source ideology
No NSFW content
No hate speech, bigotry, etc

Related Communities

Community icon from opensource.org, but we are not affiliated with them.

founded 7 years ago

MODERATORS

Cloak@lemmy.ml

kevincox@lemmy.ml

CrypticCoffee@lemmy.ml

Lettuceeatlettuce@lemmy.ml

Internet Archive to New York: Don’t Kill the Good Bots in the Fight Against Bad Bots | Internet Archive Blogs (blog.archive.org)

submitted 1 week ago by farooqkz@realbitcoin.cash to c/opensource@lemmy.ml

8 comments fedilink hide all child comments

I guess Anubis also prevents "good" bots such as search engine bots to access a webpage...

you are viewing a single comment's thread
view the rest of the comments

[–] cupcakezealot@piefed.blahaj.zone 1 points 1 week ago (1 children)

look in the crawlers directory and you can add any allows you want with the proper user agent. :)

[–] MonkderVierte@lemmy.zip 1 points 1 week ago* (last edited 1 week ago) (1 children)

Nah, that was a joke; Dillo is basicslly a graphical Lynx (both have useragent-exceptions on some big "js-only" sites). And the Anubis dev is, for a while now, working on a static version, if you believe the error message. Though that would be hard to do, if at all possible.

[–] cupcakezealot@piefed.blahaj.zone 2 points 1 week ago

ooh i'm sorry i didn't catch that <3