This is a most excellent place for technology news and articles.
Yeah, that has like 0 chances for working. At most it would annoy bots for web search, at least it has a proper robots.txt.
But any agent trying to process data for AI is not going to go to random websites. It's going to use a curated list of sites with valuable content.
At this point text generation datasets can be achieved with open data, and data sold by companies like reddit or Microsoft, they don't need to "pirate" your blog posts.
scrape.maxDepth = 5
What's stopping the sites with valuable content from using this?
I think sites that feel they have valuable content can deploy this and hope to trap and perhaps detect those bots based on how they interact with the tarpit
This is really interesting.
What a great name!
This showed up on HN recently. Several people who wrote web crawlers pointed out that this won’t even come close to working except on terribly written crawlers. Most just limit the number of pages crawled per domain based on popularity of the domain. So they’ll index all of Wikipedia but they definitely won’t crawl all 1 million pages of your unranked website expecting to find quality content.
Did you read the article? (There is a link to a non walled version.)
Since they made and deployed a proof-of-concept, Aaron B said their pages have been hit millions of times by internet-scraping bots. On a Hacker News thread, someone claiming to be an AI company CEO said a tarpit like this is easy to avoid; Aaron B told 404 Media “If that’s, true, I’ve several million lines of access log that says even Google Almighty didn’t graduate” to avoiding the trap.
Millions of hits may sound like a lot, but you need to view that in context.
What's the context?
The modern internet. Millions of hits is very normal - one of my domains is just 30 year old ASCII art of a penguin, and it gets 2-3 million a month from bots/crawlers (nearly all of them trying common exploits). The idea that the google spider would be notably negatively impacted by this is kinda naive. It could fall fully into the tarpit and it probably wouldn't even get flagged as an abnormal resource allocation. The difference in power between desktop and enterprise equipment is at this point almost inexpressible.
People think of hacking like a thief with a lockpick. It's oftentimes more like someone methodically checking every door in the neighborhood for any that are unlocked.
If it is linked to the Internet then it'll be hit by crawlers. Their "trap" isn't any how many show up but how long each bot stays on their individual site.
Can confirm, I have a website (https://2009scape.org/) with tonnes of legacy forum posts (100k+). No crawlers ever go there.
It's a shame that 404media didn't do any due diligence when writing this
Sorry to tell you, but you are indexed at least by duckduckgo, bing, ecosia, startpage, google, and even one of searx' crawlers has payed you a visit.
No crawlers ever go there.
if it makes you feel any better, i would go there if i was a web crawler.
2009scape!? If it's what I think it is that is amazing. Legend
It is what you think it is, come join ^^. It's a small niche world
Then that's a where we hide the good stuff
Reminds me of burying folders in folders in folders to hide naughty content as a youth.
Totally brilliant and foolproof. Humans can't open folders
What kinda stuff
The best stuff
I think this rate limiting mechanism is mostly a niceness rule : you should try to not put too much pressure on any website and obey the rules defined in its robots.txt.
So I guess this idea is not bad as it would mostly penalize bad players.
More accurately, it traps any web crawler, including regular search engines and benign projects like the Internet Archive. This should not be used without an allowlist for known trusted crawlers at least.
Just put the trap in a space roped off by robots.txt - any crawler that ventures there deserves being roasted.
More accurately, it traps any web crawler
More accurately, it does not trap any competent crawlers, which have per domain limits on how many pages they crawl.
You would still want to tell the crawlers that obey robots.txt do not pay attention to that part of the website. Otherwise it's just going to break your SEO
My new favorite is asking if it's cheating to look at your opponent's pieces in chess.
For anybody who ever had this happen, ChatGPT has some solutions to remedy the situation:
I tried the same input and got a more expected answer.
This reminds me of that one time a guy figured out how to make "gzip bombs" that bricked automated vuln scanners.
I had a scanner that was relentless smashing a server at work and configured one of those.
evidently it was one of our customers. it filled their storage up and increased their storage costs by like 500%.
they complained that we purposefully sabotaged their scans. when I told them I spent two weeks tracking down and confirmed their scan were causing performance issues on our infrastructure I had every right to protect the experience of all our users.
I also reminded them they were effectively DDOSing our services which I could file a request to investigate with cyber crimes division of the FBI.
they shut up, paid their bill, and didn't renew their measly $2k mrr account with us when their contract ended.
bitch ass small companies are always the biggest pita.
