Full title: I hid a link on my website that only bots can see and told them not to follow it. They broke the rules and in 4 days they went 124,414 levels deep into an endless maze. One bot opened 782,580 doors. I thought this was super interesting, this is how the internet is these days. Thought I’d share.

you are viewing a single comment's thread
view the rest of the comments
[–] 16 points 4 days ago (3 children)

Can anyone explain what is happening behind the scenes with the "room creation"?? That sounds intriguing. I understand OP is a bot but, is anything described here an actual way of trapping crawlers? How are the crawlers fooled by any of this?

  • source
  • hideshow 3 child comments
  • [–] 28 points 4 days ago (2 children)

    He placed an invisible link on his website. Only visible by reading the page content (html). Then the bots followed the link. This link lead to a page with 5 links (door) four links are dead and one leads to another page (level) with again 5 links.

    I guess the links and pages are generated on the go. So the bots and crawlers may follow the links forever of they like.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 1 day ago (1 child)

    I wonder if pages have to be generated individually to appear legit to the crawlers, or if the same contents can be served as different links over and over... if I knew more about how this stuff works I'd try it myself hah.

  • source
  • parent
  • hideshow 1 child comment