222
Why are anime catgirls blocking my access to the Linux kernel? (lock.cmpxchg8b.com)
submitted 1 year ago by to c/selfhosted@lemmy.world
117 comments fedilink hide all child comments
 

Some thoughts on how useful Anubis really is. Combined with comments I read elsewhere about scrapers starting to solve the challenges, I'm afraid Anubis will be outdated soon and we need something else.

you are viewing a single comment's thread
view the rest of the comments
[–] 21 points 1 year ago (10 children)

Anubis is no challenge like a captcha. Anubis is a ressource waster, forcing crawler to resolve a crypto challenge (basically like mining bitcoin) before being allowed in. That how it defends so well against bots, as they do not want to waste their resources on needless computing, they just cancel the page loading before it even happen, and go crawl elsewhere.

  • permalink
  • fedilink
  • source
  • hideshow 10 child comments
    •
  • [–] [S] 7 points 1 year ago (9 children)

    No, it works because the scraper bots don't have it implemented yet. Of course the companies would rather not spend additional compute resources, but their pockets are deep and some already adapted and solve the challenges.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 9 child comments
    •
  • [–] 13 points 1 year ago (5 children)

    To solve it or not do not change that they have to use more resources for crawling, which is the objective here. And by contrast, the website sees a lot less load compared to before the use of Anubis. In any case, I see it as a win.

    But despite that, it has its detractors, like any solution that becomes popular.

    But let's be honest, what are the arguments against it?
    It takes a bit longer to access for the first time? Sure, but that's not like you have to click anything or write anything.
    It executes foreign code on your machine? Literally 90% of the web does these days. Just disable JavaScript to see how many website is still functional. I'd be surprised if even a handful does.

    The only people having any advantages at not having Anubis are web crawler, be it ai bots, indexing bots, or script kiddies trying to find a vulnerable target.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 2 points 1 year ago

    I use uMatrix, which blocks js by default, so it is a bit inconvenient to have to enable js for some sites. websites which didn't need it before, which is often the reason I use them, now require javascript.

  • permalink
  • fedilink
  • source
  • parent
  • [–] [S] 1 point 1 year ago (1 child)

    Sure, I'm not arguing against Anubis! I just don't think the added compute cost is sufficient to keep them out once they adjust.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 1 year ago

    Conceptually, you could just really twist the knobs up. A human can wait to read a page for 15 seconds. But you're trying to scrape 100,000 pages and they each take 15 seconds... You can make it expensive in both power and time that's a win.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 0 points 1 year ago* (1 child)

    I'm against it for several reasons. Running unauthorized heavy duty code on your end. It's not JS in order to make your site functional, it's heavy calculations unprompted. If they would add simple button "click to run challenge" would at least be more polite and less "malware-like".

    For some old devices the challenge last over 30 seconds, I can type a captcha in less time than that.

    It blocks behind the necessity to use a browser several webs that people (like the article author) tend to browse directly from a terminal.

    It's a delusion. As shown by the article author solving the PoW challenge is not that much of an added cost. Span reduction would be the same with any other novel method, crawlers are just not prepared for it. Any prepared crawler would have no issues whatsoever. People are seeing results just because it's obscurity, not because it really works as advertised. And in fact I believe some sites are starting to get crawled aggressively despite anubis as some crawlers are already catching up with this new Anubis trend.

    Take into account that the challenge needs to be light enough so a good user can enter the website in a few seconds running the challenge on a browser engine (very inefficient). A crawler interested in your site could easily put up a solution to mine the PoW using CUDA in a GPU which would be hundreds if not thousands of times more efficient. So the balance of difficulty (still browsable for users but costly to crawl) is not feasible.

    It's not universally applicable. Imagine if all internet were behind PoW challenges. It would be like constant Bitcoin mining, a total waste of resources.

    The company behind Anubis seems more shady to me each day. They feed on anti-AI paranoia, they didn't even answer the article author valid critics when he email them, they use clearly PR language aimed to convince and please certain demographics to place their product. They are full of slogans but lack substance. I just don't trust them.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 6 points 1 year ago*

    Fair point. I do agree with the "clic to execute challenge" approach.

    For the terminal browser, it has more to do with it not respecting web standard than Anubis not working on it.

    As for old hardware, I do agree that a temporization could be good idea, if it wasn't so easy to circumvent. In such case bots would just wait in the background and resume once the timer is fullified, which would vastly decrease Anubis effectiveness as they don't uses much power to do so. There isn't really much that can be done here.

    As for the CUDA solution, that will depend on the implemented hash algorithm. Some of them (like the one used by Monero) are made to vastly more inefficient on GPU than it is on the CPU. Moreover, GPU servers are far more expensive to run than CPU ones, so the result would be the same : crawling would be more expensive.

    In any case, the best solution would be by far to make it a legal requirement to respect robot.txt, but for now the legislators prefer to look the other way.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 12 points 1 year ago* (2 children)

    The point was never that Anubis challenges are something scrapers can’t get past. The point is it’s expensive to do so.

    Some bots don’t use JavaScript and can’t solve the challenges and so they’d be blocked, but there was never any point in time where no scrapes could solve them.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] -2 points 1 year ago (1 child)

    Wait, so browsers that disable JavaScript won't be able to access those websites? Then I hate it.

    Not everyone wants unauthenticated RCE from thousands of servers around the world.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •