This is a most excellent place for technology news and articles.
This sort of thing has been a strategy for dealing with unwanted web crawlers since web crawlers were a thing. It's an arms race, though; crawlers do things to detect these "mazes" and so the maze-makers keep needing to up their game as well.
As we enter an age where AI is effectively passing the Turing Test, it's going to be tricky making traps for them that don't also ensnare the actual humans you're trying to serve pages to.
I haven't seen that episode in probably 15 years and I still remember exactly what this was.
But does running this cost the AI bot at least as much as it costs you to run?
Picking words at random from a dictionary would not be very compute intensive, the content doesn't need to be sensical
Yes, the scraper is going to mindlessly gobble up information. At best they'd expend more resources later to try and determine the value of the content but how do you do that really? Mostly I think they're hoping the good will outweigh the bad.
I would think yes. The compute needed to make a hyperlink maze is low, compared to the AI processing of the random content, which costs nearly nothing to make, but still costs the same to process as genuine content.
Am I missing something?
I'm wondering about the cost to the server's resources / bandwidth to serve up unlimited random junk also.
But kudos to the developer for making this anyway
This is my concern exactly.
This seems like a neat prank, but a potentially expensive one. Heck, if it works right you could end up with several bots stuck in your maze, perhaps dozens of hundreds. At that point bandwidth becomes my concern.
This won't work against commercial crawlers. They check page contents with something similar to a simhash and don't recrawl these pages. They also have limiters like for depth to avoid getting stuck in circular links.
You could generate random content for each new page, but you'll still eventually hit the depth limit. There are probably other rules related to content quality to limit crawling too.
True, this is an arms race situation after all. The real benefit of this is creating garbage training data that makes garbage models. So it’s not just increasing the cost of crawling, it increases the cost of stealing everybody’s shit because you need extra data quality checks. Poisoning the well.
You could theoretically use the shittiest local llm you can find to dynamically create slop for the piggies
Say it with me now: model collapse! I think this approach is especially insidious in that rather than dumping obvious nonsense into the training corpus that can then be scrubbed, it pushes the downstream LLM invisibly towards spontaneously imploding.
I suspect that there are many websites that already dynamically generate an unbounded number of pages based on the links one clicks, and that Web spiders will have needed to deal with those for as long as there have been people spidering the Web, which is going to be no later than the first Web search engines.
I'd guess that if nothing else, they cap how far they spider a site. Probably a lot more sophisticated, use heuristics to figure out which sites are more worth spending indexing resources on, as it's not just whether to spider but also the frequency with which to do so. Some parts of a site are more "valuable" than others -- for a search engine, a more desirable target for users clicking on results -- and some will update more frequently and are more-useful to re-spider at higher frequency. Google will return current news articles, yet still indexes a large portion of the content out there. They won't be doing that by simply sending GoogleBot at everything that they've indexed at a fixed frequency.
This genus named genius game is sending pain to these previous devious data devourors
