Disclaimer: I don't have a background in computer science

I recently heard about a lightweight opensource software called Anubis.

Anubis is designed to stop AI crawlers that download a lot of data to train artificial intelligence models

https://anubis.techaro.lol/

Several websites have deployed Anubis:

https://gitlab.gnome.org/GNOME

10 websites have deployed. 10 websites. Out of millions of websites.

My question is extremely simple.

If this software is so damn great, why isn't it everywhere?

Seriously. Why isn't it used on Lemmy? On Wikipedia? On CBC?

all 16 comments

sorted by: hot top controversial new old
[–] 25 points 1 year ago

Because it's not a perfect solution and other sites have other solutions in place.

  • source
  • [–] 11 points 1 year ago (5 children)

    Weird that this is getting downvotes. It's a legit question.

  • source
  • hideshow 5 child comments
  • [–] 6 points 1 year ago (2 children)

    People get sensitive when religion is mentioned!

  • source
  • parent
  • hideshow 2 child comments
  • [–] 9 points 1 year ago
    • not every website has the problem it solves
    • not everyone who does likes the solution it offers
    • web development moves fast but not that fast
  • source
  • [–] 9 points 1 year ago* (2 children)
  • [–] 7 points 1 year ago

    Yeah. It's an arms race. Any technological defense will be countered eventually. In the long run, I'm not sure a technical defense like this will be sufficient - we'll need legal defenses that are enforced.

  • source
  • parent
  • [–] 7 points 1 year ago

    It isn't so great, and it is everywhere.

  • source
  • [–] 4 points 1 year ago

    It's not great, it's actually pretty ineffective.

    https://lock.cmpxchg8b.com/anubis.html

  • source
  • [–] 3 points 1 year ago

    I believe, (far too) much of the commercial world relies on Cloudflare to solve that problem.

    And as for Wikipedia, any AI trainer worth their salt should know that they don't need to crawl it, because you can actually just download the whole Wikipedia dataset.

  • source
  • [–] 1 point 1 year ago

    10 websites have deployed.

    Source?

    Because many already have solved the Problem via cloudflare, etc.

  • source
  • [–] 0 points 1 year ago*

    It's not that great, and the creator seems to have some weird fixation on little girls.

  • source