this post was submitted on 14 Aug 2026
70 points (98.6% liked)

Technology

7167 readers
336 users here now

News community around technology, social media platforms, information technology and governmental policy surrounding it.

What doesn't fit here?

The core of the story has to be technology focused.


Post guidelines

Title formatPost title should mirror the news source title. If you don't like the title of article, look for an alternative source instead of editorializing it.
URL formatPost URL should be the original link to the article (even if paywalled) and archived copies left in the body. It allows avoiding duplicate posts when cross-posting.
[Opinion] prefixOpinion (op-ed) articles must use [Opinion] prefix before the title. Opinion articles refer to articles that their publisher doesn't explictly endorse.
Country prefixCountry prefix can be added to the title with a separator (|, :, etc.) if the news is from a local publisher who doesn't clearly mention the country.


Rules

1. English onlyTitle and associated content has to be in English.
2. Use original linkPost URL should be the original link to the article (even if paywalled) and archived copies left in the body. It allows avoiding duplicate posts when cross-posting.
3. Respectful communicationAll communication has to be respectful of differing opinions, viewpoints, and experiences.
4. InclusivityEveryone is welcome here regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
5. Ad hominem attacksAny kind of personal attacks are expressly forbidden. If you can't argue your position without attacking a person's character, you already lost the argument.
6. Off-topic tangentsStay on topic. Keep it relevant.
7. Instance rules may applyIf something is not covered by community rules, but are against lemmy.zip instance rules, they will be enforced.


Companion communities

!globalnews@lemmy.zip
!interestingshare@lemmy.zip


Icon attribution | Banner attribution


If someone is interested in moderating this community, message @brikox@lemmy.zip.

founded 2 years ago
MODERATORS
 

A year of fighting scrapers on my 1.5 million-page website. What I tried, how I failed, and what's working now.

top 14 comments
sorted by: hot top controversial new old
[–] BlameTheAntifa@lemmy.world 20 points 5 days ago (1 children)

I tried to give them a human view, but it appears that they block VPNs so I gave up.

[–] snoons@lemmy.ca 26 points 5 days ago* (last edited 5 days ago) (2 children)

It's a vibe-coder ai bro dude guy complaining about ai.

but also I have IVPN and it loaded without issue.

[–] kazerniel@piefed.zip 14 points 5 days ago* (last edited 5 days ago)

It’s a vibe-coder ai bro dude guy complaining about ai.

the irony is rich...

[–] bugbear@lemmy.sdf.org 1 points 4 days ago

thanks, you saved me time opening that link

[–] codexarcanum@lemmy.dbzer0.com 12 points 5 days ago

I work on an ecomm site and it's nightmarish. We have all the cloudflare protections turned on but do try to tune out system so american AI companies will scrape us (we want chatgpt to send us customers and use our data).

Still, we have a country allow list because it's easier to admit North American traffic and the specific IPs our contractors use, then block everyone else. The amount of bot traffic flooding in globally is insane. I don't think the current day internet infrastructure is going to be able to absorb the abuse much longer.

[–] fubarx@lemmy.world 5 points 5 days ago

I thought Cloudflare's anti-AI defenses were meant to dynamically catch all of this. That's a lot of work to do to keep bot traffic away.

FWIW, I tried loading it with a VPN, it showed the content, but not all the images. Turned off VPN, it fully loaded. Maybe the blog has different rules.

[–] melroy@kbin.melroy.org 2 points 5 days ago (3 children)

I feel you! Same issue here. I created my own solution https://angieguardian.org/. Fully open source and free to use under agpl license.

[–] undefinedTruth@lemmy.zip 3 points 4 days ago* (last edited 4 days ago) (1 children)

Why would a website owner use this over Anubis. Not bashing it or anything, just curious how it differentiates?

[–] melroy@kbin.melroy.org 2 points 4 days ago

I wanted to have high performance. See my benchmarks. I'm able to reach 150k-180k reqs/sec allowing me to handle ddos attacks as well. Even under very heavy load. Anubis will fall apart.

Great question BTW!

[–] onlinepersona@programming.dev 1 points 5 days ago (1 children)

Is that like the anime girl proof we see everywhere?

[–] melroy@kbin.melroy.org 5 points 5 days ago

No that's Anubis

[–] BrikoX@lemmy.zip 0 points 5 days ago (1 children)

I don't run PatronView, just sharing the blog post from them.

I tried to access angieguardian.org, but was met with "JavaScript is required to access this site" which basically guaranteed I will never visit it again.

[–] melroy@kbin.melroy.org 2 points 5 days ago

OK fine. That is due to the pow. The whole point of Angie Guardian. I also don't like it. But the internet is fked up!

[–] elvith@feddit.org 2 points 5 days ago

Wait.... AJ12Bot and Ahrefsbot are still around?! I have them blacklisted since... At least 2012 now, I think?

I usually don't really evaluate which bots might or might not be around anymore and still keep the rules for them, LOL