this post was submitted on 23 Jan 2025
923 points (97.7% liked)
Technology
60942 readers
5638 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
This showed up on HN recently. Several people who wrote web crawlers pointed out that this won’t even come close to working except on terribly written crawlers. Most just limit the number of pages crawled per domain based on popularity of the domain. So they’ll index all of Wikipedia but they definitely won’t crawl all 1 million pages of your unranked website expecting to find quality content.
Did you read the article? (There is a link to a non walled version.)
Millions of hits may sound like a lot, but you need to view that in context.
What's the context?
The modern internet. Millions of hits is very normal - one of my domains is just 30 year old ASCII art of a penguin, and it gets 2-3 million a month from bots/crawlers (nearly all of them trying common exploits). The idea that the google spider would be notably negatively impacted by this is kinda naive. It could fall fully into the tarpit and it probably wouldn't even get flagged as an abnormal resource allocation. The difference in power between desktop and enterprise equipment is at this point almost inexpressible.
People think of hacking like a thief with a lockpick. It's oftentimes more like someone methodically checking every door in the neighborhood for any that are unlocked.
If it is linked to the Internet then it'll be hit by crawlers. Their "trap" isn't any how many show up but how long each bot stays on their individual site.
Can confirm, I have a website (https://2009scape.org/) with tonnes of legacy forum posts (100k+). No crawlers ever go there.
It's a shame that 404media didn't do any due diligence when writing this
if it makes you feel any better, i would go there if i was a web crawler.
Sorry to tell you, but you are indexed at least by duckduckgo, bing, ecosia, startpage, google, and even one of searx' crawlers has payed you a visit.
2009scape!? If it's what I think it is that is amazing. Legend
It is what you think it is, come join ^^. It's a small niche world
Why would they? Outrage and meme content sell clicks, in-depth journalism doesn't.
I think you may have just misunderstood the post.
It's not intended to trap the web crawlers indexing content for google search.
It's intended to trap AI training bots harvesting sentences in order to improve their LLMs.
I don't really have an answer as to why those bots don't find your content appealing, but that doesn't mean that Nepenthes doesn't work.
Then that's a where we hide the good stuff
An even easier way to hide stuff is to not put it online in the first place.
Reminds me of burying folders in folders in folders to hide naughty content as a youth.
Totally brilliant and foolproof. Humans can't open folders
When I worked as a technician in a computer repair company, it was amazing the number of people that were just put that stuff on the desktop.
What kinda stuff
The best stuff
Like what?
Like stuff that is not bad.
Rule out the mediocre too, unless it’s extremely mediocre then it’s OK
Pron
I think this rate limiting mechanism is mostly a niceness rule : you should try to not put too much pressure on any website and obey the rules defined in its robots.txt.
So I guess this idea is not bad as it would mostly penalize bad players.