this post was submitted on 16 Aug 2026
91 points (96.9% liked)
Privacy
50414 readers
597 users here now
A place to discuss privacy and freedom in the digital world.
Privacy has become a very important issue in modern society, with companies and governments constantly abusing their power, more and more people are waking up to the importance of digital privacy.
In this community everyone is welcome to post links and discuss topics related to privacy.
Some Rules
- Posting a link to a website containing tracking isn't great, if contents of the website are behind a paywall maybe copy them into the post
- Don't promote proprietary software
- Try to keep things on topic
- If you have a question, please try searching for previous discussions, maybe it has already been answered
- Reposts are fine, but should have at least a couple of weeks in between so that the post can reach a new audience
- Be nice :)
Related communities
much thanks to @gary_host_laptop for the logo design :)
founded 6 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
and how do you propose to impose that? Heck robots.txt doesn't even seem to be honored!
Eh. I make web pages for friends and family. We don't want them indexed. In my server logs, most of the big scrapers are pretty good about honoring robots.txt. Google does. Meta does. At least some the AI scrapers do. Pretty sure Bytedance does too but I'd have to double check.
I'll see a hit from let's say G's bot, fetching robots.txt, and then no other activity. For all the thigns I can shit on G and Meta for, this isn't one of them.
Is it universal to heed it? Nah, ofc not. But it's still more effective than you'd think, given it is voluntary.