Some thoughts on how useful Anubis really is. Combined with comments I read elsewhere about scrapers starting to solve the challenges, I'm afraid Anubis will be outdated soon and we need something else.

you are viewing a single comment's thread
view the rest of the comments
[+] -13 points 1 year ago* (last edited 1 year ago) (26 children)

Yeah, well-written stuff. I think Anubis will come and go. This beautifully demonstrates and, best of all, quantifies the negligence negligible cost to scrapers of Anubis.

It's very interesting to try to think of what would work, even conceptually. Some sort of purely client-side captcha type of thing perhaps. I keep thinking about it in half-assed ways for minutes at a time.

Maybe something that scrambles the characters of the site according to some random "offset" of some sort, e.g maybe randomly selecting a modulus size and an offset to cycle them, or even just a good ol' cipher. And the "captcha" consists of a slider that adjusts the offset. You as the viewer know it's solved when the text becomes something sensical - so there's no need for the client code to store a readable key that could be used to auto-undo the scrambling. You could maybe even have some values of the slider randomly chosen to produce English text if the scrapers got smart enough to check for legibility (not sure how to hide which slider positions would be these red herring ones though) - which could maybe be enough to trick the scraper into picking up junk text sometimes.

  • source
  • hideshow 26 child comments
  • [–] 4 points 1 year ago (18 children)

    @mfed1122 @tofu any client-side tech to avoid (some of the) bots is bound to, as its popularity grows, be either circumvented by the bot’s developers or the model behind the bot will have picked up enough to solve it

    I don’t see how any of these are going to do better than a short term patch

  • source
  • parent
  • hideshow 18 child comments
  • [–] 1 point 1 year ago (1 child)

    Yeah, you're absolutely right and I agree. So then do we have to resign the situation to being an eternal back-and-forth of just developing random new challenges every time the scrapers adapt to them? Like antibiotics for viruses? Maybe that is the way it is. And honestly that's what I suspect. But Anubis feels so clever and so close to something that would work. The concept of making it about a cost that adds up, so that it intrinsically only effects massive processes significantly, is really smart...since it's not about coming up with a challenge a computer can't complete, but just a challenge that makes it economically not worth it to complete. But it's disappointing to see that, at least with the current wait times, it doesn't seem like it will cost enough to dissuade scrapers. And worse, the cost is so low that it seems like making the cost significant to the scrapers will require really insufferable wait times for users.

  • source
  • parent
  • hideshow 1 child comment
  • @mfed1122 yeah that is my worry, what’s an acceptable wait time for users? A tenth of a second is usually not noticeable to a human, but is it useful in this context? What about half a second, etc

    I don’t know that I want a web where everything is artificially slowed by a full second for each document

  • source
  • parent
  • load more comments (16 replies)
  • load more comments (23 replies)